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Abstract. We study the problem of existence of a periodic point in the 
boundary of an invariant domain for a surface homeomorphism. In the area- 
preserving setting, a complete classification is given in terms of rationality of 
Caratheordory's prime ends rotation number, similar to Poincare's theory for 
circle homeomorphisms. In particular, we prove the converse of a classic result 
of Cartwright and Littlewood. This has a number of consequences for generic 
area preserving surface diffcomorphisms. For instance, we extend previous 
results of J. Mather on the boundary of invariant open sets for C-generic 
area preserving diffeomorphisms. Most results are proved in a general context, 
for homeomorphisms of arbitrary surfaces with a weak nonwandering-type hy- 
pothesis. This allows us to prove a conjecture of R. Walker about co-basin 
boundaries, and it also has applications in holomorphic dynamics. 



The boundary of a simply connected domain on a surface can have a very com- 
plicated topology. A classic example of this fact is the pseudo-circle |Bin48[ |Han82j. 
which is a hereditarily indecomposable continuum, and in particular nowhere locally 
connected. For this reason, there is a very sharp step between studying the dynam- 
ics of a circle homeomorphism and studying the dynamics of a homeomorphism on 
the boundary of an invariant simply connected domain. 

On the circle, Poincare introduced the notion of rotation number, an invariant 
for the dynamics, and established the following celebrated result: 

There is a periodic point if and only if the rotation number is rational. 
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In order to study the dynamics of a surface homeomorphism / on the boundary 
of a simply connected domain U, it is natural to try to define a similar invariant 
and replicate Poincare's results. This can be done by means of Caratheodory's 
prime ends compactihcation [Carl3a, Carl3b. Mat82 which is obtained by adding 
a circle to U with an appropriate topology so that the resulting space U = U U S 1 
is homeomorphic to the closed unit disk D. The homeomorphism / extends to a 
homeomorphism / of U, therefore inducing a homeomorphism /|§i on the circle 
of prime ends. One may in this way associate to / and U a prime ends rotation 
number p(f, U) as the Poincare rotation number of /|§i. 

It is already noted in |CL51j that a rational p(f, U) does not necessarily imply 
that there is a periodic point in d U , and neither the converse is true. However, 
in the area-preserving setting, the situation is different, as is shown by Cartwright 
and Littlewood in [CL51J : 

If f is area-preserving, U is relatively compact, and the prime ends 
rotation number is rational, then there is a periodic point in dU . 

Similar results, showing that the rationality of the rotation number leads to the 
existence of periodic points with the area-preserving hypotheses replaced by weaker 
conditions near the boundary of U are also studied in [CL51J . Since then, several 
contributions were made in the same direction, e.g. [AY92I [BG92, BG9L BK98, 
IQRdPllj . 

However, little is known about the converse problem. Namely, if p(f, U) is 
irrational, what can we say about the dynamics on d U1 This question is already 
mentioned in [CL51], and simple examples show that even in the area-preserving 



case there can be a fixed point in dU (see Examples 9.5 and 9.6). However, the 
examples are very specific and have exactly one fixed point and no other periodic 
points. 

This is the subject of this paper. We present a complete classification theorem 
in this direction, which also gives topological information on the boundary of U 
in the case that the rotation number is irrational. Such a classification has appli- 
cations both in the context of C"*-generic area preserving diffeomorphisms, and in 
holomorphic dynamics. 

1.1. Main results. Note that if / is area- preserving on a closed surface, then / is 
nonwandering (i.e. every nonempty open set intersects one of its forward iterates). 
Since in general only the latter (weaker) condition is used, we state our results for 
nonwandering maps. 

All the examples of a nonwandering homeomorphism of a closed surface where 
p{f, U) is irrational and there is a periodic point in dU are on the sphere, and they 
all have the property that d U is a non-separating continuum with exactly one fixed 
point and no other periodic point. Such examples exist even in the C°° category 



(see Example 9.5 1. The next theorem shows that this is the only possibility, and 
provides additional topological information (see Section [2] for definitions). 

Theorem A. Let f ': S — > S be a nonwandering orientation-preserving homeomor- 
phism of a closed orientable surface, and U an open f -invariant simply connected 
set whose boundary has more than one point and such that p{f, U) is irrational. 
Then one of the following holds: 

(i) dU is a contractible annular continuum without periodic points; 
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(ii) S is a sphere, U is dense in S, and S \ U is a non-separating continuum 
containing a unique fixed point and no other periodic points. 

We would like to emphasize that without any assumption of nonwandering type 
this result is not valid, as shown by an example of Walker [Wal9l] (see Example 



9.1). However, since most of the arguments are local in a neighborhood of the 



boundary of U, it is possible to relax the nonwandering condition to a much weaker 



one. We call it the d- nonwandering condition (see Definition 3.101. Wc will not 
define it here, but one should keep in mind that it is weaker than requiring that 
some neighborhood of the boundary of U be contained in the nonwandering set of 



/ (see Proposition 3.14), and in particular it holds if / is area-preserving. But in 
addition, this condition holds in other contexts, for instance when / is a holomorphic 
diffeomorphism with irrational rotation number. Most of our results are proved 
under the 9-nonwandering hypothesis. 

We also mention that in case (i) of TheoremjAj the rotation number of any invari- 
ant probability measure supported in d U (defined using an annular neighborhood) 
is equal to p(f, U) (see |Fra88|lMatl2] ). 

The proof of Theorem [5] is based on the following 

Theorem B. Let f : R 2 — > R 2 be an orientation preserving homeomorphism and 
U C R 2 be an open f -invariant simply connected set. If f is d -nonwandering in 
U and p(f, U) ^ ; then there are no fixed points for f on the boundary of U . 
Moreover, if U is unbounded then there are no fixed points for f in the complement 
ofU. 

In fact, using the above theorem we obtain a more general version of Theorem 
|A} which holds for surfaces which are not necessarily compact: 

Theorem C. Let f ': S — > S be an orientation preserving homeomorphism of an 
orientable surface S of finite type, and U C S an open f -invariant topological disk 
such that S\U has more than one point. Assume further that f is d -nonwandering 
in U and p(f, U) is irrational. Then exactly one of the following holds: 

(i) dU is aperiodic, and U is compact and contractible; 

(ii) S is a sphere, and the only periodic point of f in dU is a unique fixed point; 

(iii) S is a plane, and dU is unbounded and aperiodic. 

Moreover, there is a neighborhood W of dU , which can be chosen as an annulus in 
case (i), a disk in case (ii) and the complement of a closed disk in case (iii), such 
that every connected component of S \dU contained in W is wandering. 

A main ingredient in the proofs is the following technical theorem, which has its 
own character. It relates the genus of the surface, the prime ends rotation number 
of an invariant open disk, and the existence of a translation arc intersecting its 
boundary. 

An arc 7 is called TV-translation arc for a homeomorphism /, if it is a simple arc 
connecting a point x to its image f{x) and the concatenation of its first N iterates 
is also a simple arc. 

Theorem D. For any g e N U {0} and O^ae R/Z, there is N a , g <E N such that 
the following holds. Let S be an orientable surface of genus g, and f : S — > S an 
orientation preserving homeomorphism. Suppose that U C S is an invariant open 
topological disk such that S \ U has more than one point, f is d -nonwandering in 
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U, and p(f,U) — a. Then there is a compact set K C U such that every N a>g - 
translation arc in S\K is disjoint from d U . 

For a motivation on this result, see the introduction of Section |4j We only 
mention here that a typical situation where N- translation arcs appear is when one 
considers the stable or unstable manifold of a fixed point of saddle type. 

The prime ends compactification can be defined for more general (non-simply 
connected) sets. Theorems |A| [C and [d] are generalized to this context as Corollary 



7.21 Theorem 17. 11 and Theorem 



7.3| respectively. 

1.2. Consequences for area-preserving surface diffeomorphisms. The above 
results have a number of consequences on generic properties of area-preserving sur- 
face diffeomorphisms. See Section [H] Here, we mention two of them. 

Theorem E. Let f be a C r generic area preserving diffeomorphism on a closed 
orientable surface, r > 1. Then there are no periodic points on the boundary of any 
periodic complementary domain. 

Recall that a complementary domain is a connected component of the comple- 
ment of a continuum (i.e. of a compact connected set with more than one point). 
The generic condition in this theorem is given explicitly in Section [8] 

This theorem solves a problem that was extensively studied by J. Mather |Mat81j . 
He proved that the (Caratheodory) rotation number of any topological end of a com- 
plementary domain U is irrational under certain conditions (the so-called Moser 
genericity) which hold generically in any C r topology if r is sufficiently large (e.g. 
r > 16). This gave some information on the boundary of U , for instance implying 
that any periodic point in d U is necessarily a hyperbolic saddle p with its stable 
and unstable manifolds also contained in d U. It is known that this is not possi- 
ble if the surface is the sphere or the torus, due to results of Pixton [Pix82| and 
Oliveira OH87J which imply that p would have a homoclinic intersection. However, 
for surfaces of higher genus, analogous results are not known (in fact, they are false 
under the specific generic conditions of Mather's Theorems), and therefore it is not 
possible to rule out the existence of the point p € d U by the same methods. 

The use of Theorems [A] and [D] provide a novel approach to the same problem, 
allowing for a proof that does not require the (generic) existence of homoclinic 
intersections and thus allowing to obtain a proof on surfaces of any genus. 

Let us remark that Theorem [E| was mistakenly used in |Xia06] and |KN10| (see 



{ 8.5 and Remark 8.8 ), and therefore it fills a gap in the respective articles (however, 
the gap in jKNlfj] can be avoided by using a different result; see |KNllj ). 

1.3. Other consequences. The 9-nonwandering condition also holds whenever 



the dynamics induced in the circle of prime ends is transitive (see £3.4 1. This ob- 
servation, together with Theorem [C] and a result from |BG91j . leads to the following 
result which proves a conjecture of Walker jWa!91| : 

Theorem F. Suppose f : S 2 — > S 2 is an orientation-preserving homeomorphism, 
and K C M 2 is a continuum that irreducibly separates the plane into exactly two 
invariant components. If the dynamics induced in the prime ends from one side of 
K is conjugate to an irrational rotation by angle a, then the prime ends rotation 
number from the opposite side of K is also equal to a. 
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Indeed, if the rotation numbers differ, then the main theorem from BG91 im- 
plies that there are many periodic points in K 7 but since K is the boundary of a 
9-nonwandering invariant domain with irrational rotation number, this contradicts 
Theorem [Cl 

Theorem |F] proves Conjecture 2 of |Wal91| . namely that the prime ends dynamics 
from the two sides of a cobasin boundary (i.e. a set such as K) cannot correspond 
to two non-conjugate irrational rotations. 

Another context in which similar ideas can be applied is in the setting of holomor- 
phic maps. Suppose U C C and / is a homeomorphism defined in a neighborhood 
of U and holomorphic in U, leaving U invariant, and such that p{j, U) is irrational. 
By means of the Riemann mapping theorem and the Schwartz refleaction principle, 
one may show that the homeomorphism induced by f\jj on the circle of prime ends 
is conjugate to that of an analytic diffeomorphism [PM97j. This means that the 
dynamics cannot be Denjoy type, implying automatically that the 9-nonwandering 
condition holds. Thus, one may apply our results in this setting. In particular, one 
may obtain purely topological proofs of some of the results from Ris99 . 

1.4. Extension of the theory of prime ends. Another result introduced in this 
paper, which is of independent interest, is a natural extension of the theory of 
prime ends. In Section [3j a precise description is given of what happens with the 
prime ends compactification of an open set when of the underling surface changes. 
Although this forces us to introduce some cumbersome notations, it is necessary to 
our proof, and it allows us to consider the prime ends compactification of sets which 
are not relatively compact (as opposed to what is usually done in the literature). 

1.5. Structure of the article. This article is organized as follows: Section [2] in- 
troduces preliminary notations and definitions, and some general results that will 
be used later. In Section [3] we formally define the notion of prime ends compactifi- 
cation for arbitrary simply connected sets, and we establish results analogous to the 
ones in |Mat81j and |Mat82| about the topology of prime ends. The proofs are given 
in the Appendix to avoid bloat. We also define the prime ends rotation number and 
establish certain invariance of this number when the underlying surface is changed. 
In j ]3.4| we define precisely the notion of 9-nonwandering, and we show that it is 
weaker than other conditions, for instance / being nonwandering or preserving a 



finite area. In {3.5 we prove the result of Cartwright and Littlewood stated in the 
beginning of the introduction, but under the 9-nonwandering condition. 

Section [4] is devoted to the main technical result of the article, which is Theorem 
|D| The proof is by contradiction, and the main idea is that if the theorem does 
not hold one can construct a large enough disjoint pairs of loops with intersection 
index ±1 (contradicting the finite genus). 

Sectionjsjcontains a proof of Theorem |B| The main idea of the proof is to combine 
Theorem |D] with arguments from Brouwer theory which, together with a theorem 
of Jaulent on maximal unlinked sets, lead to the existence of translation arcs near 
a fixed point under certain conditions. 

In Section [6] we prove Theorems [A] and [C] and a version of the same theorems 
for arbitrary surfaces of finite genus is also stated. The proof is based on applying 
Theorem [B] on a lift of / to the universal covering; however additional nontrivial 
arguments are required to deal with the fact that / is not necessarily nomotopic to 
the identity, and to prove the contractibility of the boundary of U. 



6 



ANDRES KOROPECKI, PATRICE LE CALVEZ, AND MEYSAM NASSIRI 



Section [7] generalizes the results from Section [6] to an arbitrary open set U: 
assuming there is a regular end (or ideal boundary point, as defined in [2| of U 
which is fixed by / and has irrational prime ends rotation number, one obtains 
information about the impression of the given end analogous to the one given for 
the boundary of U in the simply connected case. The main idea of the proof 
consists in modifying the surface to reduce the problem to one in which U is simply 
connected. Theorem iDl is also stated in this context. 

Section p\ is devoted to applications of the main results of this article, proving 



Theorem [El which is in fact a simple corollary of the more general Theorem 8.3 



(see ^8.3). Other applications are also mentioned. 

Finally, the examples in Section [9] show that the theorems fail without a d- 
nonwandering condition. We also exhibit examples with an irrational prime ends 
rotation number but exhibiting a fixed point (on the sphere), and an example 
showing that the number N a . g depends on both a and g in Theorem ID] 



2. Preliminaries 

Let S be a surface (which is always assumed Hausdorff, connected and ori- 
cntable). When U C S and it is clear that S is the underlying surface, we denote 
by U the closure of U and d U the boundary of U in S. If W C S and U C W, we 
write cl\v U and d\y U for the closure and boundary (respectively) of U in W with 
the restricted topology. 

A set is called an open (resp. closed) topological disk if it is homeomorphic to 
the open (resp. closed) unit disk in M 2 . Similarly, an open (resp. closed) topological 
annulus is a set homeomorphic to the annulus S 1 x (0, 1) (resp. S 1 x [0, 1]). 

We consider an arc 7 to be a continuous map from an interval in K to S, and 
7 is called a simple arc if it is inject ive, except possibly at its endpoints. If 7 is 
an arc, we abuse notation and also denote its image by 7; it should be clear from 
the context which one is the case. We say that 7 : [a, b] — > S is a loop if its initial 
point and its end point coincide. We say that a loop is essential in S if it is not 
nomotopic (with fixed endpoints) to a point, and an open set U C S is essential in 
S if it contains some loop which is essential in S. 

A translation arc (or 1-translation arc) is a compact simple arc in S joining a 
non-fixed point x to its image f(x), and such that f(-f) n 7 = {f(x)} or {x, f{x)} 
(where the latter case holds only when f 2 (x) = x). More generally, an N -translation 
arc is a translation arc such that, additionally, / fe (7)H7 = for 2 < k < N— 1, and 
7^(7) H 7 is either empty or {x} (the latter holding only when f N+1 (x) — x). This 
is equivalent to saying that the path Y\ 0<k<N f k {l) obtained by concatenation is 
still a simple arc. If 7 is an A^-translation arc for every N > 1, we say that 7 is an 
00 -translation arc. This is equivalent to saying that one of the arcs (and so every 
arc ) ]lfc<o/ fc (7), ri/c>o/~ fe (7), or rifcez/ fe (7) is a simple arc. 

Let us recall a classic result of Brouwer (see, for instance, |Fat87j ). 

Proposition 2.1 (Brouwer's lemma on translation arcs). Any translation arc of 
a fixed-point free orientation preserving homeomorphism of Mr is an 00 -translation 
arc. 

2.1. Continua. A continuum is a compact connected set with more than one point. 
We say that the continuum K C S is 
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• cellular if there is a sequence (D n ) ne jq of closed topological disks such that 
int(D„ + i) C D n and K = f|„ eN D n ; 

• annular if there is a sequence {A n ) nl z^ of closed topological annuli such 
that A n+ i is contained in the interior of A n and separates its boundary 
components, and K — HneN^™' 

• contractible in S if there is a continuous map H : K x [0, 1] —> S and xq G S 
such that H (x, 0) = x and iT(x, 1) = xo for all x £ K. 

Let us note the following equivalent properties. 

• K is contractible in S if and only if it has a neighborhood U C S homeo- 
morphic to an open disk; 

• K is cellular if and only if it is contractible and non-separating; 

• K is annular if and only if it has a neighborhood U homeomorphic to an 
open annulus such that U \ K has exactly two components, both essential 
in U. 

2.2. Topological ends (ideal boundary points). If S is a non-compact surface, 
a boundary representatiue of S is a sequence Pi D Pi D ■ ■ ■ of connected unbounded 
(i.e. not relatively compact) open sets in S such that d$ P n is compact for each n 
and for any compact set K C S, there is n > such that P n n K — if n > n 
(here we denote by ds P n the boundary of P n in S) . Two boundary representatives 
(Pi)ieN an d (Pj')iGN are said to be equivalent if for any n > there is to > such 
that P m C P^, and vice- versa. The ideal boundary h\S of S is defined as the set 
of all equivalence classes of boundary representatives. Denote by ci S the space 
S U bi S with the topology generated by open sets of S together with sets of the 
form VUV', where V is an open set in S such that ds V is compact, and V denotes 
the set of elements of bi S which have some boundary representative (Pj)ieN such 
that Pi C V for all i € N. We call ci S the ends compactification or ideal completion 
of S. 

Any homeomorphism / : S — > S extends to a homeomorphism / : ci S — ¥ Cj S 
such that /|s = /. If 5" is orientable and has finite genus, then ci S is a closed 
orientable surface of the same genus, and bi S is a totally disconnected closed set. 
Sec [Ric63j for details. 

If U is an open subset of a surface S, and p € bi U is an ideal boundary point, 
we define its impression in S by 

Z(p) = f){c\ s (V nU):V open in ci U, p G V}. 

Note that Z(p) C 5 S U. 

Assuming S has finite genus and U d S, we say that p is a regular ideal boundary 
point of Z7 C S if p is isolated in bi U and Z(p) has more than one point. 

Given a regular end p of U, we say that A c C/ is a p-collar if A U {p} C ci /7 is 
a closed topological disk. 

2.3. Some general results. Let us state a result that allows us to make 'surgery' 
arguments in several places. It can be seen as a refined version of Schoenflies' 
theorem. Its proof can be found in |Eps66| §2]. We give a simplified statement, 
since we consider boundaryless surfaces only. 

Theorem 2.2 (Epstein, jEps66 ). Let 71, 72: S 1 — > S be homotopic simple loops in 
a surface S on which none of the two loops bounds a disk. Then there is an isotopy 
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(ft*)te[o,i] from the identity on S to some map h — hi such that h o ji = 727 o,nd 
the isotopy is fixed (pointwise) outside some compact set. 

We also recall a useful extension theorem, a proof of which can be found in 
[LR041 Theorem 2.1] (see also [Ham54j ). 

Theorem 2.3 (Extension Theorem). Let W be a connected open subset ofM 2 and 
H : W — > W C K 2 a homeomorphism with a unique fixed point zq . Then there is a 
homeomorphism H' : R 2 — > K 2 which coincides with H in a neighborhood of z and 
such that Zq is the unique fixed point of H' . 

Finally, we state a result for future reference. By an aperiodic set we mean one 
which does not contain a periodic point. 

Theorem 2.4 ( |KorlOj ). If f : S — > S is a nonwandering homeomorphism of a 
closed orientable surface and K C S is an aperiodic invariant continuum, then K 
is annular. 

3. Prime ends compactifications 

From now on, whenever we say that S is a surface this will mean that S is a 
connected orientable surface of finite genus. 

Let us recall some facts and definitions from Caratheodory's prime ends theory. 
For more details of the classic theory, we refer the reader to |Mat82j and [Mat81j: 
however, we will need some work to generalize the theory to our context and to find 
a relation between prime ends compactifications on different spaces. In particular, 
we define prime ends for non-relatively compact sets and obtain the classic results 
in that context. 

Let U C S be a simply connected open subset of the surface S such that S\U 
has more than one point. An end-cut of U in S is an arc 7: [0,1) — > U such 
that lim^i j(t) — x for some x € dU (so that 7 extends continuously to an arc 
[0, 1] U with 7(1) = x E dU). A point z € dU is accessible (from U) if it is the 
endpoint of some end-cut in U. 

A cross-cut of U in S is the image of a simple arc 7: (0, 1) — > U that extends 
to an arc 7: [0, 1] — > U joining two points of dU, and such that each of the two 
components of U \ 7 has some boundary point in d U \ 7. Note that the endpoints 
of a cross-cut are accessible points. A cross-section of U in S is any connected 
component of U \ 7 for some cross-section 7 of U in S. Note that each cross-cut 
corresponds to exactly two cross-sections, which are topological disks. 

A chain for U in S is a sequence C = (D n ) ne ^ of cross-sections such that Di C D.j 
for all i > j > 1 and djj Di n du Dj = for all i ^ j. If D is any cross-section 
of U, we say that the chain C divides D if Di C D for all sufficiently large i. If 
C = (D' n ) e fs is another chain, we say that C divides C if for each n > there is 
m such that D m C D' n . We say that C and C are equivalent if C divides C and C 
divides C. A chain C is called a prime chain if C divides C whenever C is a chain 
that divides C. An equivalence class of prime chains is called a prime end of U. 

Remark 3.1. Our definition of prime end is basically the same as the one in [Mat82] . 
except that we use cross-sections instead of arbitrary topological disks. This makes 
no difference in practice, since one can (as Mather does) easily show that a chain of 
the more general type is always divided by a chain as defined here, and vice-versa. 



PRIME ENDS ROTATION NUMBERS AND PERIODIC POINTS 



9 



Moreover, in [Mat82j it is required that U be relatively compact, but this will not 
be necessary in our case. 

Denote by bs{U) — bg(Z7, S) the set of all prime ends of U, and cs(U) = 
cs(U,S) — U U bf(J7). For a cross-section D of U, we say that the prime end 
p € bs(U) divides D if some (hence any) chain representing p divides D. Denoting 
by £u,s(D) the set of all prime ends that divide D, we can topologize ce(U) by 
defining a basis of open sets consisting of all sets of the form DU £u t g(D) for some 
cross-section D of U, together with all open subsets of U. 

Remark 3.2. Note that even though a cross-section is a subset of U, its definition 
depends not only on U but also on the surface S: if S' C S is an open set containing 
U, some cross-sections of U as a subset of S may cease to be cross-sections when U 
is regarded as a subset of S' (this happens when an endpoint of the corresponding 
cross-cut lies outside 5"). This is why we use the cumbersome notation cs(U,S) 
when there is need to emphasize what the underlying surface is. 

The next theorem gives a useful relation between ce(U,S) and ce(U,S'), while 
it also extends what is known in the relatively compact case to the general case. 
Since its proof is somewhat lengthy and not central to this article, we present its 
proof in the Appendix. 

Theorem 3.3 (Prime ends compactification). The following properties hold: 

(1) bs(U,S) is homeomorphic to a circle S 1 , and cs(U,S) is homeomorphic to 
the closed disk D. 

(2) Let So C S be an open connected set such that [/ C S . Then the inclu- 
sion i : U — > cs(U,So) extends to a monotone surjection i* : cs(U,S) — » 
c £ (U,S ). 

(3) If S\Sq is totally disconnected, then the inclusion i: U — > bf(f7, So) extends 
to a homeomorphism i* : cs{U, S) — > cs(U, So). 

3.1. Accessible prime ends and criteria for primality. Let us introduce the 
following notation: if (ifj)jgN is a sequence of sets, then Ki — > x £ S as i —¥ oo if 
for each neighborhood V of x there is io such that Ki C V for all I > io- We recall 
a useful criterion for "primality" in the case that U is relatively compact in S. The 
results are contained in [Mat811 Lemmas 3.1-3.4 and Proposition 3.7]. 

Proposition 3.4. If U is relatively compact in S, then 

(1) A chain of U in S is prime if and only if there is x € dsU and some 
equivalent chain (Z>j)j E N such that djj Di — > x as i — > oo; 

(2) // 7: [0, 1) — > U is an end-cut of U in S, then there is a prime end p £ 
bg(U, S) such that j(t) — > p in C£(U, S) as t — > 1~ . 



Parts (1) and (3) of Theorem 3.3 imply the following version of the previous 



result in the case that U is not relatively compact: 

Proposition 3.5. The following properties hold: 

(1) A chain of U in S is prime if and only if there is x € C\(S) and some 
equivalent chain (Di)i £ N ofU in S such that d\j D t — > x in Ci(S) as i — > 00; 

(2) // 7: [0, 1) — > U is an end-cut of U in S, then there is a prime end p £ 
bf (17, S) such that j(t) — ?> p in cs(U, S) as t — > 1~ . 
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We say that p G bf (U,S) is an accessible prime end if there is an end-cut 
7: [0, 1) — > U such that j(t) — > p in Cf(f7, 5) as i 1~. In this case, the limit 
z = lim^i- 7(t) in 5 does not depend on the choice of 7, but only on the choice of p. 
Indeed, if (-Di)igN is any chain of U in S representing p and such that djj Di — > x G 
Ci(S) as i — ¥ 00 (given by part (1) of the previous proposition), since 7 accumulates 
in p in cs(U,S), it must intersect any neighborhood of p in cs(U,S) and so 7 
intersects for each i G N. This implies that 7 intersects 3y if i is large 
enough, and since djj Di — > x we conclude that a; is a limit point of 7, so z = x. 
But a; depends only on the chosen cross-section and not on the choice of 7; thus we 
conclude that there is a unique possible value of z (and of x) for the given p. 

Therefore, any accessible prime end has an associated accessible point in dU 
which is uniquely determined. Similarly, by part (2) of the previous proposition, 
every accessible point z G dU gives rise to at least one (but possibly more than 
one) accessible prime end, namely the endpoint in cg({7) of any end-cut of U which 
converges to z in S. 

Note that the previous remarks imply in particular that if 7 is a cross-cut of U 
in S, then cl cs ms) 1 consists of 7 together with two points of hg(U, S). The next 
proposition summarizes some useful properties of prime ends; we omit the proofs 
since they are simple consequences of the definitions and the previous propositions. 

Proposition 3.6. Let D be a cross-section of U in S bounded by the cross-cut 7 
in U . 

(1) D contains some end-cut 77 : [0, 1) — > D with endpoint in d U \ 7. 

(2) Let a, b G hs{U 1 S) be the endpoints 0/7 in cs(U,S). Then the endpoint of 
1] in Cs(U,S) as t — > 1~ is a prime end p that divides D and is different 
from a and b. 

(3) a ^ b, and £u t g(D) is a nonempty open interval of the circle bs(U,S) 
bounded by a and b. 

(4) Accessible prime ends are dense in bg(t/, S). 

(5) Accessible points of dU are dense in dU. 

3.2. Dynamics and rotation numbers for prime ends. If /: S —> S is an 

orientation-preserving homcomorphism such that f(U) — U, then f\u maps cross- 
sections to cross-sections, and it extends to a homeomorphism /*: cs(U,S) — > 



Cs(U, S). The next result, which is a direct consequence of Theorem 3.3 says that 
if one considers a smaller connected surface So such that U C Sq C S , then the map 
induced by / in cs(U, S) is semi-conjugate to the map induced by / in cs(U, Sq). 

Corollary 3.7. If So is an open connected set such that U C Sq C S and /(So) = 
So, and if f* and fl denote the extensions of f\u to Cs(U, S) and c.g(U,So), re- 
spectively, then = f'J,*, where i*: cs(U,S) —t Cs(U,Sq) is the monotone map 



extending the inclusion i : U <— > cs(U, Sq) given by Theorem 3.3 



Indeed, note that if\u — f\ui holds trivially, so the previous result for the 
extensions follows by continuity. 

Remark 3.8. A simple example of the previous corollary is a map / : M 2 — > M 2 that 
leaves the unit disk invariant and such that /|§i has two fixed points: an attractor 
N and a repellor S. If So = S \ K, where K is one of the two closed intervals of 
S 1 limited by N and S, then f\u induces on the circle of prime ends \>s(U, R 2 ) the 
same dynamics of /|gi . However, the map induced by f\jj on bf(C/, Sq) corresponds 
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to a circle homeomorphism with a unique fixed point, which is a saddle-node. The 
semi-conjugation collapses one of the intervals between fixed points to a single 
point. 

If /* is the homeomorphism of cs(U,S) induced by /|;y, then /*|b £ (;y,s) is a 
homeomorphism of the circle, and we may consider its rotation number in the 
sense of Poincare, which we denote by p(f, U, S) £ K./Z (or by p(f, U) when there 
is no ambiguity). If two circle homeomorphisms are monotonically semi-conjugate, 
then they have the same rotation number. Thus from the previous corollary we 
obtain the following crucial result. 

Corollary 3.9. If So a S is open, connected, f -invariant, and U C So, then 
p(f,U,S) = p(f,U,S ). 

3.3. Prime ends for non-simply connected sets. Suppose that U C S is an 
open connected set but not necessarily simply connected, and let p G bi U be a 
regular ideal boundary point of U. We may define a p-cross-cut as any simple arc 
7 contained in a p-collar which extends to a compact arc 7 joining two different 
points of Z(p), and such that each of the two components of U \ 7 contains some 
point of Z(p) \ 7 in its boundary. If 7 is a p-cross-cut, then at least one of the two 
components of U \ 7 is contained in some p-collar. Any such component is called a 
p-cross-section. Define p-chain and prime p-chain using the same definitions as in 
the previous sections, except that using p-cross-sections; and define a p-prime end 
as an equivalence class of prime p-chains. 

We denote the family of all p-prime ends by bf p U, and we set c £ p U = U Ub £p U. 
As before, the topology on C£ p U is defined by using as a basis of open sets the open 
sets of U together with sets of the form V U £ p V, where V is a p-cross-section of U 
and £p V is the set of all p-prime ends which have a representative chain entirely 
contained in V. Note that if p <E ci U and if A is a p-collar, then every p-prime end 
has a representative p-chain entirely contained in A. 

If U has finitely many ideal boundary points, all of which are regular, then one 
can do the above process simultaneously for each end of U obtaining a compactifi- 
cation which we denote by c £ U and is a compact surface with boundary. We use 
the notation b £ U = c £ U \ U = \J pehl v b £p U. 

The results from the previous section can be naturally extended to this setting. 
In particular, bf p U is homeomorphic to S 1 for each regular ideal boundary point 
p. To avoid further technicality, we will only make use of this fact in the case that 
U is relatively compact, and for such case this is proved in [Mat81j and |Mat82j . 

If / : S — > S is an orientation preserving homeomorphism and p is a regular ideal 
boundary point of U fixed by /, then f\u extends to a homeomorphism /* of C£ p U, 
and we may define the rotation number p(f, p) as the classical rotation number of 
the circle homeomorphism /*|b £p u- 

3.4. The 9-nonwandering condition. The main hypothesis for many of our re- 



sults is the following (see Proposition 3.14 for simpler properties that imply this, 



for example the preservation of a finite area). 

Definition 3.10. Let /: S —> S be a homeomorphism and E/cSa simply con- 
nected open /-invariant set. We will say U is d-nonwandering for / (or / is d- 
nonwandering in U) if, for every n € N, there is some compact set K C U such 
that U\K does not contain the forward or backward /"-orbit of any /"-wandering 
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cross-section. In other words, if D is a cross-section of U such that f nk (D) n D = 
for all k 0, then f nk (D) intersects K for some positive and some negative value 
of ft. 

Remark 3.11. It can be assumed that the set K is a closed topological disk, by 
increasing its size. We will make this assumption most of the time. 

Remark 3.12. We do not know if Definition |3.10| using only n = 1 implies the 
definition as stated here. However, in view of Proposition |3 . 1 4| this does not make 
a difference in the usual setting (e.g. nonwandering or area-preserving maps). 

We can extend the definition to the non-simply connected setting as well. 

Definition 3.13. If U C S is a connected open /-invariant set and p € bi U a fixed 
regular end of U. We will say / is d -nonwandering at p if for each n G N there is a 
p-collar A C U such that A does not contain the forward or backward orbit of any 
/""-wandering p-cross-section. 

The next proposition implies, in particular, that / is d- nonwandering in U if f\u 
is nonwandering. 

Proposition 3.14. Let f: S — > S be a homeomorphism and U C S a simply 
connected open f -invariant set. Then f is d -nonwandering, provided that there is 
a compact set K C U such that any one of the following holds: 

(1) U \ K does not contain the forward or backward orbit of any f -wandering 
open set. 

(2) There is an f -invariant Borel measure (not necessarily finite) such that 
fj,(U \ K) < oo and fJ-(D) > for any cross-section D, or 

(3) U\K is contained in the nonwandering set of f. 

(4) The dynamics induced by f\u on h£(U,S) is transitive. 

Proof. Suppose that (1) holds and / is not 9-nonwandering. Then there is n G N 
such that any neighborhood oidU contains the forward or the backward /"-orbit of 
some /"-wandering cross-section. Let K' be a compact set such that K C f k (K') C 
U for all ft G Z with |ft| < n. Then we may assume that there is a cross-section Do 
of U such that f kn (D) flD = (J for all nonzero ft G Z, and f kn (D ) CU\K' for 
all integers k > (the case k < is analogous). 

Note that our choice of K' implies that f k (Do) c U \ K for all k > 0, so by 
(1) we must have f k °{D ) nfl ^l for some k G N. Let D x = f ka (D„) n Do- 
Again (1) implies that f kl (Di) n D\ ^ for some k\ G N. Inductively, we may 
define fco, k\,...,k n and a sequence ^ D„ +1 C D„ ■ ■ ■ C Dx C D , such that 
Di+\ — f ki {Di) n Di. We can find < i < j < n such that, if ^ = J2m=o ^m, then 
l.- L = lj (modn), so that lj — U + mn for some integer m > 0. But it follows from 
our definition that Dj C f lj ~ li {Di), and since Dj and Di are both subsets of Do 
we conclude that / m "(Do) = / ij_ < (Do) intersects Do, contradicting the fact that 
f km (D ) n D = for all k > 0. 

If (2) holds, fix n G N and let D be an /"-wandering cross-section, then {f kn (D) : 
k > 0} is an infinite family of pairwise disjoint sets of the same positive measure, 
so they cannot be all contained in U \ K, which has finite measure. Similarly for 
{f kn (D) : k < 0}, proving that / is 9-nonwandering in U. 

If (3) holds then (1) clearly holds. Finally, if (4) holds, then the induced dynamics 
by /" on bs(U,S) is minimal for any n G N. This implies that there are no /"- 
wandering cross-sections at all, so / is 9-nonwandering in U. □ 
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The next result will also be helpful in most of our proofs. 

Proposition 3.15. Let f: S — > S be an orientation-preserving homeomorphism 
of an orientable surface S of finite genus, U C S an open invariant topological 
disk, and K C S \U is a closed f -invariant set such that dU \ K 7^ 0. If f is 
d-nonwandering in U , then f\s\K * s a ^ so d-nonwandering in U . If, in addition, K 
is totally disconnected, then the converse also holds. 

Proof. The first implication is direct, since a wandering cross-section of U in S \ K 
is also a wandering cross-section of U in S. For the converse, it suffices to show that 
if K is totally disconnected, then any wandering cross-section D of U in S contains 



some cross-section of U in S\K. But this follows from part (3) of Theorem 3.3 □ 



3.5. Cartwright-Littlewood result for rational rotation numbers. This the- 
orem is essentially proved in [CL51j , but the proof is on the sphere and the hypothe- 
ses are slightly different. 

Theorem 3.16 (Cartwright-Littlewood). Let f: S — > S be an orientation-preserving 
homeomorphism of a surface S of finite genus, and U C S an open, connected, rel- 
atively compact invariant set with a regular end p fixed by f. If p(f,p) — and f 
is d-nonwandering at p, then there is a fixed point in Z(p). 

Proof. First note that since U is relatively compact, els U — cl Cl s U, so that we 
may assume that S is closed by working with ci S instead of S. 

Let (-Di)igN be a chain representing a fixed prime end in b^ p U . By Proposition 
|3.4| we may choose it such that d Di — > x for some x € dU. We will show that x 
is a fixed point. 

Since the chain represents a fixed prime end, for each j € N there is i > j 
such that f(Di) C Dj. Note that D. L C Dj as well, because i > j. We claim 
that du Dj intersects f~ 1 {duDj) if j is large enough. Suppose on the contrary 
that dDj n f~ 1 (djj Dj) = 0- Since f^ 1 (Dj) and Dj both contain Di, there are 
two cases: either Di C f~ 1 (c\ u Dj) C Dj or Di C clu(Dj) C f~ 1 (Dj). Suppose 
without loss of generality that Di C f~ 1 (c\jjDj) C Dj. By our assumption, the 
cross-cut du Dj has no endpoint in common with f~ l (du Dj). From this, it is easy 
to obtain a cross-section E C Dj \ f~ 1 {Dj) (as in Figured), and it follows that 
f~ n (E) C Dj for all n E N. Since Dj lies inside any given p-collar if j is large 
enough, this contradicts the 9-nonwandering condition at p. 

We have showed that f~ 1 (dDj) n dDj ^ for all sufficiently large j. Since 
d Dj — > x, we conclude that / _1 (.t) = x, completing the proof. □ 

4. A LEMMA ON TRANSLATION ARCS 

The main result of this section is motivated by the following simple result. 

Proposition 4.1. Let S be a closed surface, f : S — > S an area-preserving home- 
omorphism and U C S an open simply connected f -invariant set such that S \ U 
has more than one point. Then there is no 00 -translation arc intersecting both U 
and its complement. 

Proof. If there is an oo-translation arc 7 intersecting U and S \ U, then there is 
some sub-arc a C 7 U f(j) that is a cross-cut of U. If D\ and Di are the two 
components of U \ a, the preservation of area implies that the homeomorphism 
(x,y) H> (f(x),f(y)) of S x S preserves a measure of full support, and therefore 
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the set D\ x D 2 is nonwandering for such map. This implies that there is n > 
such that f n (Di) n D x ^ and f n (D 2 ) n D 2 ^ 0, which in turn implies that 
f n (cr) n a 0, contradicting the fact that a U /"(c) is contained in the simple arc 
Jfcez . 



LW fc (7)- □ 



We observe that in this proposition it is essential that the translation arc 7 
intersects both U and dU. For example, if U is a disk bounded by the stable 
manifold of a fixed hyperbolic saddle with a homoclinic connection, any simple arc 
in the homoclinic connection joining a point to its image gives an oo-translation 
arc contained in d U . The idea of the next result is to remove the hypothesis that 
7 intersects U, by requiring that the rotation number on the prime ends circle 
of U be nonzero. To do that, we prove a finite version of the previous result, 
namely that for some N > there is no iV-translation arc intersecting both U 
and its complement. Note that any small perturbation of an N- translation arc 
supported on the complement of a neighborhood of the endpoints of the arc is still 
an N- translation arc, a fact that is no longer true for oo-translation arcs. This is 
why having a version of the previous proposition for ./V-translation arcs (with the 
restriction that the rotation number is nonzero) allows us to remove the condition 
that 7 intersects U. 

The proof of the finite version of the previous result is much more delicate, and 
it is necessary to take into account both the prime ends rotation number and the 
genus of the surface in the choice of N. 

Additionally, one can prove the previous result using not the preservation of area 
but the fact that the nonwandering set of / contains U. We will relax this condition 
even more by requiring a weak type of nonwandering condition near the boundary 



of U, namely the 9-nonwandering condition (see Definition 3.10). 



The main theorem of this section is Theorem |D] in the introduction: 

Theorem 4.2. Let S be an orientable surface of genus g < 00, f:S—¥ S an 
orientation preserving homeomorphism, and U C S an invariant open topological 
disk such that S \ U has more than one point. Suppose that f is d -nonwandering 
in U , and p(f, U) = a =/= 0. Then there is N a-g G N, depending only on a and g, 
and there is a compact set K C U such that every N ' ag -translation arc in S\K is 
disjoint from d U . 

Remark 4.3. Let us emphasize that N a ^ g necessarily depends on both a and g. See 
examples |9.3| and |9.4| in Section [9] 

Before proceeding to the proof, we state a direct but useful consequence. 



Corollary 4.4. Under the hypotheses of Theorem \4-S?\ every oo-translation arc in 
S \ K is disjoint from d U . 



Remark 4.5. It follows from the proof of Theorem 4.2 that, if one replace the d- 
nonwandering condition by the condition that there are no wandering cross-sections 
(which holds for instance if / preserves area and U has finite area), then there is 
no iV a , 9 -translation arc intersecting dU at all (i.e. K = 0). 

4.1. Maximal cross-cut lemma. We begin with a technical lemma before moving 
to the proof of Theorem |4.2| 
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Lemma 4.6. Let f : S — > S be a homeomorphism and U C S a simply connected 
open /-invariant set such that S\U has more than one point and p(f, U) 7^ 0. Sup- 
pose N > 2, and 7 is an N -translation arc with endpoints in dU and intersecting 
U, and let T = ljf =0 f k {l)- Then 

(1) The set T of all connected components ofUHT consists of cross-cuts of U 
such that each a € T is disjoint from its image by f , and exactly one com- 
ponent ofU\a is disjoint from its image by f. We denote this component 
by D a , and its closure in U by D a = D a U a. 

(2) There is a unique connected component Uo of U \ T which is not disjoint 
from its image by f . Moreover, U \ Uo = [J cre jrD rJ , and Uq is open and 
simply connected. 

(3) The boundary of U in U is a union of elements of a subfamily F* C J ' , 
and each point of du Uo is accessible from Uo . 

(4) Suppose additionally that T C V, where V is a neighborhood of dU such 
that U\V is connected and intersects its image by f , and V does not contain 
the forward or backward orbit of any wandering cross-section of U . Then 
there is an arc a C 7 such that f k {<j) € J-* for all k with < k < N. 




Figure 1. The set U 



Proof. (1) Let Jb be the family of all connected components of U H 7. Since 7 
is a translation arc, f("f) can only intersect 7 in its endpoints (which are in dU). 
Thus, f(a) is disjoint from a for any a € Tq. Noting that dU is invariant and 
T = {f k {o~) ■ o € J"o) < k < N}, it follows that f(a) is disjoint from a for any 
a 6 J. But then, one of the two components of U \ a is disjoint from the image 
of a. We denote that component by D a . It follows easily that the image of D a 
either contains D a or is disjoint from D a : indeed, the connected set D a must be 
contained in one of the two connected components of U \ /(c), and one of them is 
f{D a ). But Du C /(Ax) is not possible since p(f, U) 7^ 0. Thus f{D a ) is disjoint 
from D a . Letting D a = a U D a = cl(y D CT , it follows that U \ D a is not disjoint 
from its image. This proves (1). 

(2) We now define a partial order in T as follows: a -< a' if D a C D a i. Note 
that whenever it makes sense, / preserves the order. 

Claim 1. If a and a' are different elements of T and neither a < a' nor a' -< 0, 
then D a is disjoint from D a i . 
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Proof. The connected set U \ D a is not included in D a > because this last set is 
disjoint from its image by / and the first one is not. It is not included in U\D a i by 
hypothesis (otherwise one would have a' -< a). One deduces that er'n (U\D a ) ^ 0. 
The boundary of U \ D a in U being c, which is disjoint from tr', one deduces that 
a' H D a = 0. This implies that the connected set D a must be contained in one of 
the two connected components of U\o~'. As it is not included in D a i (otherwise one 
would have a -< a') it is contained in the other one. In other words, it is disjoint 
from D a i. □ 

Claim 2. For each neighborhood VofdU there are at most finitely many elements 
of J- that are not contained in V. Moreover, there are finitely many a G J- such 
that D a is not contained in V . 

Proof. It follows from the fact that T is an embedded compact arc and elements 
of T are connected open subsets in the restricted topology of T: the set F \ V is 
compact, and the elements of J- that are not contained in V form an open covering 
of r \ V (in the restricted topology of T) by pairwise disjoint sets, so that there are 
finitely many such sets. 

To prove the second claim, note that one may choose a neighborhood V C V 
of dU such that K — U \ V is connected and f(K) intersects K. If a G T and 
a C V , then one of the components of U \ a contains K. But since D a is disjoint 
from its image, it follows that D a is disjoint from K, so it must be contained in V. 
Thus there arc at most finitely many a E T such that D a <f_ V. □ 

Claim 3. Every element of T is contained in a maximal element. 

Proof. Suppose not. Then one can find a sequence {cr n } n >o of elements of T such 
that a n -< a n +i for each n > 0. Let V be a neighborhood of d U such that D a<) (f_ V. 
One deduces that D an <f_ V for any n > 0. This contradicts Claim [2j □ 

Denote by F* the family of maximal elements of and let Uq = U\{J ae:F D a = 
U \ {J ae jr, D a . Note that if a and a' are different elements of J 7 *, then neither 
a < a' nor a' ~< a, so Claim [l] implies that {D a : a € J-*} are pairwise disjoint, 
and from Claim [2] one sees that the union of those sets is closed in U. This shows 
that Uq is open. The connectedness and simply connectedness follows easily from 
the fact that D a is a simply connected unbounded (i.e. not relatively compact) set 
in U . This proves (2). 

(3) Let o"o G J~* and x e <to- From Claim [2j if B is a small enough neighborhood 
of x, there are at most finitely many elements <7i, ...,<j/. of J 7 intersecting B and 
different from o~$. Since the cross-cuts &i are closed in U and pairwise disjoint, 
there is a smaller neighborhood B' of x that is disjoint from <j\ U- • • U<jfc. Hence B' 
is disjoint from all elements of T except ctoj an d since <7o is an embedded arc, there 
is a smaller neighborhood B" c B' such that Co O B" is a connected embedded arc 
separating B" into exactly two components B\ and B^ (and therefore x is accessible 
from each component). In particular B\ and B2 are connected and disjoint from 
Uo-gJ 7 * a 1 anc ^ both djj B\ and djj B2 contain x. Suppose B\ is the component that 
intersects D ao (and thus it is contained in D<t ). We claim that B2 C Uq. Indeed, if 
B2 intersects D a for some a G J-*, then B2 C D a , and a 7^ oq. Thus T) a is disjoint 
from D ao , contradicting the fact that x G c\(j Bi n cl^ B 2 C D a n -Do- . Thus, i?2 is 
disjoint from D a for all a G J 7 *, implying that B 2 C Uq. 
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We have showed that any point of an element er G J 7 * is accessible from Uq. This 
also shows that Uo-e.^* a c Uq. But as we already saw, Uo-e.^* D a is closed, so its 
boundary (and hence the boundary of its complement Uq) is contained in Uo-e^* a i 
and we conclude that Uo-e.F* a = ®u Uq as claimed. 

(4) Denote by Fk those elements of J 7 of the form f k (cr) with a £ Fq, and 
T* k = T k C\T*. Thus J 7 = (j£L ^ and = ll£=o ■ 

Claim 4. If a £ F'^ with < k < N, then either /(cr) £ or f(a) -< a' for 

some a' £ J 7 ^. 

Similarly, if a £ J~l with < k < N , then either f~ 1 (a) £ "F^—i or / _1 (cr) -< a' 
for some cr' £ 

Proof. We only prove the first claim; the other is proved with a similar argument. 

Suppose < k < N and a £ J2. If /(cr) is not maximal, then /(cr) -< a' for 
some a' £ J 7 *, where < i < N. Suppose that i > 0. Then / _1 (cr') £ J 7 , and so 
a = / _1 (/(ct)) -< / _1 ((t'), which contradicts the maximality of cr. Thus the only 
possibility is i = 0, as required. □ 

Claim 5. Either J" * ^ or T* N ^ 0. 

Proof. From Claim [3] we know that J 7 * is nonempty, which means that there exists 
some maximal element. So some J 7 ^ is nonempty, and the previous claim then 
implies that either J-q is nonempty or J 7 ^ is nonempty. □ 

Claim 6. Suppose there is no a £ Fq such that f k {a) £ T k for all < k < N . If 
J * ^0 (resp. J 7 ^ y^tt), then any neighborhood V of dU containing T contains the 
forward (resp. backward) orbit of some cross-section of U . 

Proof. We assume J B ' / 8 (the other case is analogous, using f^ 1 instead of /). 
For tr e J ', let n a be the first positive integer such that f n "(cr) ^ J 7 *. From our 
assumption, we have that 1 < n a < N for each a £ Tq. Moreover, from Claim [4] 
we must have (cr) -< cr' for some a' £ J-q . Let 

C = {f k (a) : a £ J" *, < k < n a } C J 7 *. 

If W = Uo-ec ^ CT > n °t m g that this is a disjoint union, it follows easily from Claim 
[2] that du W — Uaec a - Let us see tnat f( w ) c W: Any c G C is of the form a = 
f k (cr ) for some cr £ Fq and < k < n aa . If k < n CTo -l, then f(a) = f k+1 (a ) £ C, 
so that f(D a ) = Df(a) C W; and if k = n ao — 1 then Claim [4] implies that there is 
a' £J 'cC such that f(a) = f n °° (a ) -< cr', so that f{D a ) = D f(<j) C D a , C W. 
In both cases, /(Ax) C VF, showing that f(W) C VF. 

We also claim that &u f N (W) C W. To see this, note that if a £ Tq then 
/""(cr) C and thus f n (a) C VF for any n > n a . Since every element of C is a 
positive iterate of some element of J 7 ^, it follows that f N (cr) c for all er G C. 
Recalling that djjW = {J aeC cr, we conclude that f N (clu W) cWas claimed. 

We will find a wandering cross-section of U contained in W. To do this, fix 
it £ J-q, and suppose D a is nonwandering (otherwise there is nothing to do). Then 
there is a smallest integer n > such that f n (D a ) intersects D a . Since f n (D a ) C 
W, there is cr € C such that f n {D rT ) C D CT0 , and so _D CT intersects Z? CTo . But then 
from Claim [T] and the maximality of Gq and cr, we conclude that a — o-q. Thus 
f n (D a ) C 



18 ANDRES KOROPECKI, PATRICE LE CALVEZ, AND MEYSAM NASSIRI 

Let us show that n > n a : if n < n a , then /"(cr) € J-* for all < k < n, and 
together with the fact that Dtni a \ — /"(A) C Ax, this implies that /"(cr) = cr. 
But then f k (cr) € J 7 * for all k G Z, contradicting our hypothesis. Thus n > n CT . 

Note that from the fact that 7 is an TV-translation arc with N > 2 follows that 
two different elements of T cannot share their two endpoints. Thus, if a' is the 
element of J-q such that f n "(cr) -< cr', we know that /""(cr) and a' do not share 
their two endpoints, and /"(Ax) C /"-"-(Ax')- Denote cr" = f n - n "(a') and 
D(jf = f n - n '{D a ,). Since /"(Ax) C Da" and / n (Ax) intersects Ax, it follows 
that Ax" intersects Ax- Since Ax is a connected component of W and Ax" C W, 
it follows that 

/"(Ax) c Ax" c Ax. 




Figure 2. Obtaining the wandering cross-section E 

This implies that Ax \ f n (D a ) is a wandering set, so to find a wandering cross- 
section of U in W it suffices to show that Ax \ /"(Ax) contains some cross-section 
E of U. This will be a consequence of the fact that /"(cr) and a" do not share 
some endpoint (see Figure [2]): Let z G 9 J7 be an cndpoint of /"(cr) that is not 
an endpoint of a". Then we can choose a small disk D around z that is disjoint 
from a" and from the remaining endpoint of /"(cr). Since D necessarily intersects 
Ax" \ /"(Ax) and it does not contain U, the boundary of D contains a simple arc 
P joining a point x G dU to a point j/ G /"(cr) and such that (3 C Ax" \ /"(Ax)- 
The concatenation of /3 with the sub- arc of /"(cr) joining y to z gives a cross-cut /?' 
of [/ such that f3' c Ax" \ /"(Ax)- A simple connectedness argument shows that 
one of the cross-sections E defined by j3' in U is contained in Ax" \ / n (Ax)> as we 
wanted. 

Finally, let V be a neighborhood oi dU such that Z7" \ V is connected and not 
disjoint from its image by /, and suppose T C V. If 7 G J 7 , then 7 is disjoint from 
[/ \ V and so J7 \ V is contained in one component of U \ 7. Since A is disjoint 
from its image, it follows that U \ V is disjoint from D 1 . Since this is true for all 
7 G J 7 , in particular {/ \ V is disjoint from W, i.e. W C V. Thus the wandering 
cross-section that we found earlier is contained in V. This proves the claim. □ 



Part (4) of the lemma follows directly from Claim [6j completing the proof of the 
lemma. □ 
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4.2. Proof of Theorem 

we are not interested in t 
For every £ € 



4.2 Let us begin by explaining what is iV QjS . Note that 



lis proof in searching the optimal value of this constant. 
Z write ||f|| = inf t+z=c \t\. We will set N a . g = (2g + l)q + 2, where 
the integer q is chosen as the smallest positive integer such that 



J || gall = if a is rational, 

1 (2g + l)||ga|| < ||a|| if a is irrational. 

This choice of q guarantees the following: 

Claim. If F is an orientation preserving homeomorphism o/R/Z of rotation num- 
ber a and if is an orbit of F that is not periodic, then one can choose an 
orientation o/R/Z such that the sequence 

Co) 

is cyclically ordered in the sense that the open intervals defined by two consecutive 
terms are nonempty and pairwise disjoint. 

This claim should be clear in the irrational case, since the fact that a is irrational 
implies by a classic result of Poincare that F is monotonically semiconjugate to the 
rigid rotation by a, so that the relative ordering of an orbit of F in the circle 
coincides with the relative ordering of points of an orbit of the rigid rotation by 
a (and so the condition on q easily implies that the claim holds using the usual 
orientation of the circle; see Figure EJ. 






■ «»_ (2g + l)qa + a 



F" +1 {x) 






t qa + a 



<{2g + l)qa 



Figure 3. Rational case 



Figure 4. Irrational case 



On the other hand, if a = p/q is rational (with q € N smallest possible), then 
every periodic point has period q and the non-periodic point x = £o is such that 
F kq {x) converges monotonically to a periodic point z as k — > oo. In particular, 
since q > 2, the claim is easily seen to hold, by choosing the orientation of the 
circle depending on the direction of convergence of f kq (x) to z (i.e. from the left or 
from the right). See Figure [3] 



Let us suppose that the hypotheses of Theorem 4.2 are satisfied, and let K C U 



be a compact set such that U \ Kq does not contain the forward or backward orbit 
of any wandering cross-section of U. Choose a larger compact set K such that 
K C f n {K) C K for < n < N ag . We may further assume that K is connected 
and not disjoint from its image. 
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From now on, we assume that there exists an iV-translation arc 7 in U \ K 
such that 7 meets dU, where N > N a g , and we will derive a contradiction. Let us 
consider the simple arc T = Uo<n<JV f k {l)- Observe that every sub-arc 7' of Y that 
joins a point x G 7 to f(x) is an (N — l)-translation arc. By hypothesis, one may 
choose 7' such that it meets dU at a point x' distinct from its endpoints. If I? is a 
topological disk that contains x' in its interior and that is sufficiently small, then 
/™(7' UD)fl (7' U D) \ {f(x)} = for every positive n < N - 1. This implies that 
every arc 7" that joins x to f(x) and is included in 7' U D is an N — 1 translation 
arc. As x' belongs to dU, one may construct such an arc 7" that contains x' but 
also meets U inside D. Let us consider Y" = Uo<n<iV-i f k (l")- The sub-arc of 
r" that joins x' to f(x') is an (TV — 2)-translation arc 7"' that meets U and whose 
ends are on dU. Let us define Y'" = Uo<n<Af-2 f k {l"')- 



Let us apply Lemma 4.6 to F'" by keeping the notations. Note that the hypoth- 
esis of part (4) of that lemma holds using V — U \ K and N — 2 instead of N. Thus 
one may find a cross-cut a E F* such that f k (a) 6 F* for all k with < k < N — 2. 

Since a is a cross-cut of U , its closure in C£(U) is a U {£o> Vo}> where £o ; Vo are 
prime ends of U. Let / be the extension of f\u to C£(U). We claim that £0 is 
not /-periodic. To see this, assume that £0 is periodic, and let q be the smallest 
positive integer such that / 9 (£o) = Co- Then a = p(f\b £ (u)) — p/g(niod Z) for 
some p€Z, and so by our definition, N > 7V Q g = (2g + l)q + 2 > q + 2. Since £ 
is an accessible prime end, it has a corresponding accessible point £ in d U. The 
fact that / 9 (£o) = £0 implies that f q (Co) = £o- But £ is an endpoint of cr, which is 
contained in the compact arc 7"', so £ € 7"'. Since 7"' is an (N — 2)-translation 
arc and 2 < q < N — 2, it follows that f q W) n 7"' = 0. This contradicts the fact 
that £ e cr. 

Thus £0 is not periodic, and the claim at the beginning of the proof applied to 
F = f\b e (u) implies that if ^ = / fc (£o), the sequence (jlj) is cyclically ordered in the 
circle hg(U) (using an appropriate orientation). If U$ C U is the set from Lemma 
4.6 then p(cr) C d Uq for < i < N — 2. Using these facts (and noting that 



N — 2 > g), one may choose two sequences of arcs (aj)o<z<g and , (&i)o<i<g with 
the following properties (see Figure [5]) : 

• cii and bi are in U$ except for their endpoints; 

• en joins a point of f 2iq {a) to a point of f( 2i+1 ^(a) n U; 

• b z joins a point of f 2tq+1 {<r) to a point of fi 2i + 1 )i+ 1 (a) n [7; 

• a, PI a 3 - = = bi (1 6j ifi^j, and n 6j = for any i, j; 

The arc T'" is linearly ordered by its orientation. Recall that a is a sub-arc of 
7"', so that cr lies between x' and /(a;') in the ordering of Y'" , and the sequence of 
arcs 

(a, /(a), /«+V), . . . , fVs + V«(a), j^+^+V)) 

are pairwise disjoint and increasingly ordered as sub-arcs of F'"; furthermore, they 
are all oriented in the same way (either positively or negatively) with respect to the 
orientation of Y'" . We may close to a simple loop o" by letting a" = c^Ua-, where 
a[ C r'" is a simple arc joining the endpoint of to its starting point (oriented 
accordingly). Similarly, we define a simple loop 6" = bi U b[ where b\ C Y'" is a 
simple arc joining the endpoint of bi to its initial point and oriented accordingly. 
The loops a ' , . . . , a'g are pairwise disjoint, and so are the loops 6 ' , . . . , b'g. Moreover, 
if i ^ j then a'! n W = 0, while ^aj'fl b'( C T"' and ^ a\ n 6< C r'". 
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Figure 6. The arcs a, and bi viewed in P 



Note that by construction, the intersection index of Oi with V" at the initial 
point of cii is opposite to the corresponding index at the final point of a^, and 
does not intersect V" anywhere else (and similarly with bi). This means that in a 
neighborhood of V" , the curves a, and &, lie on the same side of T (see Figure [6]). 
From these facts one concludes that the algebraic intersection number of a" and 6" 
is if i 7^ j and ±1 if i — j. This implies that one can construct g + 1 pairwise 
disjoint 1-punctured tori To, • • • > T g such that a" U b 1 ! C Tj for each i, contradicting 
the fact that S has genus g. 

A more algebraic way of saying the same thing is to note that, if [a"] and \b'(] de- 
notes the respective homology classes of a" and b'( in Hi(S, Q) and [a"]A[6f] denotes 
the intersection pairing, then [a'-] A is if i ^ j and ±1 if i = j , while [a"] A [a"] = 
= [b'f] A [b'!} for < t < j < g. It follows that {[a' '], . . . , [a^'J, [6' '], . . . , [^']} is lin- 
early independent in Hi(S,Q), concluding that dim H % (S, Q) > 2g + 2. But since 
<? is the genus of S and 5 is orientable, one should have dimH±(S, Q) = 2g. This 
contradiction completes the proof of the theorem. □ 

5. Converse of a theorem of Cartwright and Littlewood 
The main theorem of this section is the following: 
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Theorem 5.1. Let f: R 2 -> M 2 be an orientation preserving homeomorphism and 
U C M 2 an open f -invariant simply connected set. If f is d-nonwandering in U 
and p(f, U) 7^ 0, then there is no fixed point of f in the boundary of U . 

Before proceeding to the details, we mention an easy consequence: 

Corollary 5.2. Under the hypotheses of the above theorem, ifU is unbounded then 
Fix(/)CC7. 

Note that the two results above are precisely Theorem [B] in the introduction. 

5.1. Maximal unlinked sets and Brouwer theory. Let us state some results 
and definitions that we need to prove Theorem |5.1| 

If / : S — > S is a homeomorphism isotopic to the identity of an orientable surface, 
let us say that a set X c Fix(/) is unlinked if X is closed and f\s\x '■ S\X —> S\X 
is isotopic to the identity. Wc say that X is maximal unlinked if it is unlinked and 
there is no unlinked set of fixed points containing X properly. 

Remark 5.3. If S — M 2 and / preserves orientation, then any set consisting of two or 
less fixed points of / is unlinked. To see this, note that every orientation preserving 
homeomorphism of ]R 2 is isotopic to the identity [Kne26] . and let (/t)te[o,i] be an 
isotopy from Id to /. For every fixed point z, one gets an isotopy (/ t ')te[o,il that 
fixes z, by defining f[ = gt ° ft, where gt is the affine translation that sends ft{z) 
on z. If z' is another fixed point, one gets an isotopy (/ t ")te[o,i] that fixes z and z' 
by defining f" = h t o / ( , where h t is the affine map that sends ft{z) on z and ft(z') 
on z' . 

The next theorem is due to O. Jaulent. We only state the part that we will use. 

Theorem 5.4. |Jaul2j Let S be an orientable surface, f : S — > S a homeomorphism 
isotopic to the identity, and Xq is an unlinked set of fixed points of f . There exists 
a maximal unlinked set X C Fix(/) containing Xq. Moreover, if (/t)tg[o,i] is an 
isotopy from Idg\Y to f\s\x an d z G Fi x (/) \ X, then the loop (ft(z))te[o,i] * s 
homotopically nontrivial in S \ X . 

We will also use the following classic lemma. 

Lemma 5.5. Let h: K 2 — > M 2 be an orientation preserving homeomorphism, and 
7o a simple arc joining a point z to its image h{z) and containing no fixed point of 
h. Then any neighborhood of 70 contains a translation arc 7 for h such that z G 7. 

Proof. Let V be a given neighborhood of 70. Reducing V, we may assume that it 
is simply connected and contains no fixed point of h. Choose a homeomorphism 
(f>: D -> V such that 0(0) = z. For t G [0, 1] define V t = <j>(B t (0)), where B t (0) is 
the open disk of radius t centered at the origin. If 70 is not already a translation 
arc, we may choose r to be the smallest number such that V T intersects h{V T ). 
Clearly < r < 1, and V T n h(V T ) = 0. This means that there is w G d V T such that 
h(w) G dV T , and moreover w ^ h(w). Since V T is homeomorphic to a disk and 
z G , we can choose a simple arc 7 in V T joining w to h(w) and going through z. 
It is clear from the defintion that ^1(7) H 7 C {w,h(w)}, which means that 7 is a 
translation arc. □ 

The next key lemma is a consequence of Brouwer's lemma on translation arcs 
(see, for instance, |Fat87j ): 
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Lemma 5.6. Let h: R 2 -)• R 2 be an orientation preserving homeomorphism, and 
7 a translation arc for h. Then for each N G N, either 7 is an N -translation arc, 
or IJ^q h k (j) contains a loop that is homotopically nontrivial in R 2 \ Fix(/i). 

Proof. Suppose 7: [0, 1] — > R 2 joins x to h(x), let W be the connected component 
of M \ Fix(/i) containing x. Since 7 is a translation arc, it is disjoint from Fix(/), 
so it is contained in W. This implies that W is invariant. Let tt: W —> W be its 
universal covering map. Choose x € tt^ 1 (x), let 7 be the lift of 7 with 7(0) = x, 
and let h : W — > W be the lift of h\w such that h(x) = 7(1). 

Then clearly 7 is a t ranslation arc for h, and since W ~ R 2 and h has no fixed 



points, Proposition 



2.1 



implies that 7 is an oo-translation arc for h. Let T„ be the 
, h (7) . If 7r is injective on Tjv , then it is clear that 7 



concatenation of 7, ^1(7), 
is an iV-translation arc for h. Otherwise, there is a sub-arc a C F„ that joins two 
different points of tt^ 1 (z), for some z £ W. This implies that Tr(a) is a loop that is 
homotopically nontrivial in W (and thus in E 2 \ Fix(/i), as claimed). □ 

5.2. Accessible fixed prime ends. Before proceeding to the proof of Theorem 
|5.1| in its general form, we state a result which does not require the 9-nonwandering 
condition. 

Proposition 5.7. Let f : R 2 -> R 2 be an orientation preserving homeomorphism 
and U C R 2 a simply connected f -invariant set. If there is an accessible fixed point 
in d U, then p(f, U) = 0. 

Remark 5.8. The proposition above is essentially Corollary 2 of |CL51j . if one 
translates the statement to their setting. Here, we present a short direct proof for 
the sake of completeness. 

Proof of Proposition \5. 7| Suppose there is an accessible fixed point z% € dU, so 
there is an arc 7: [0, 1) — > U such that j(t) — > Z\ in R 2 as t — > Let Z\ E C£(U) 
be the prime end such that ^(t) — > z\ in Cf(C/) as t — > and denote by / be 
the extension of f\u to cs(U). Assume for contradiction that p(f,U) ^ 0. Then 
f(zi) 7^ z%. Letting j'(t) = /(7(f)) we obtain an arc in U such that j'(t) — > z% in 
M 2 and 7'(t) —t /(Si) 7^ 2i in cs(U) as i -> 1~. By reducing 7, we may assume 
that 7 and 7' are disjoint, and by joining the initial point of 7 to the initial point 
of 7' by a simple arc rj in U that is disjoint from 7 and 7' except at its endpoints, 
we obtain a simple arc a = 7 U 77 U 7' in t/ that extends in cs(U) to an arc joining 
?i to f(zi). See Figure [Tj 




7' = K7) 



b £ (U) 



I fk {i) 



FIGURE 7. View in c £ {U) (left) and K 2 (right) 
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Let D C K 2 be the open disk bounded by a U {z±}. The boundary of D n U 
in Cg(U) consists of a together with one of the two open intervals I of bf(Z7) 
determined by the points z\ and f(z±) (moreover, D is a neighborhood of I in 
bs(U)). Since I is a fundamental domain of the map /|b £ (!7)> which has nonzero 
rotation number, it follows that there is N > such that for any prime end z £ 
bf (U) there is an integer k such that < k < N and either f (z) £ I or f k (z) — z\. 

Let z 7^ z\ £ d U be an accessible point, and z an accessible prime end associated 
to z, so that there is an arc a: [0, 1) — > U such that a(t) — > z in M 2 and a(t) —> z 
in C£ U as t — >• 1~. Then as we just mentioned, there is fc with < k < N such 
that either f k (z) £ I or f k (z) = z%. Let us first show that the latter case does 
not hold: if f k (z) = Z\, since the accessible point associated to z\ is precisely z\, it 
follows that f k (a(t)) — > z\ (in IR 2 ) as f ^ 1~, and since z\ is fixed by / it follows 
that a(t) — > z\ as t — > 1~ , contradicting the assumption that z ^ z\. 

Thus we must have j k {z) £ I. Since f k {a(t)) = f k (a(t)) £ D if t < 1 and 
is close enough to 1, it follows that f k (z), which is the limit of f k (a(t)) in K 2 
as t — > 1~ , belongs to D n dU. Moreover, since f k {z) ^ Zi, and z\ is the only 
element of d DDdU, we conclude that f k (z) £ D. Since < k < N, it follows that 

z £ UfcLo/"*^)- Let W = Uk=of~H D )- We just showed that every accessible 
point of dU other than z\ belongs to W . Since accessible points are dense in dU, 
it follows that d U C W. Thus, if O denotes the unbounded connected component 
of R 2 \ W, we conclude that U is disjoint from O. But dO C dW C U U {zi} 
(because d D C U U {^i}), and clearly dO ^ {^i}, so J7 fl dO ^ 0, contradicting 
the fact that U D O — 0. This completes the proof. □ 

5.3. Proof of Theorem |5.1[ The proof is by contradiction. Assume from now on 
that there is a fixed point Z\ £ d U. We begin by reducing the problem to a simpler 
setting. 

Claim 1. We may assume that f has a unique fixed point zq in U . 

Proof. To see this, we will find a map /' that coincides with / in a neighborhood 
of d U (and hence satisfies the hypotheses of the theorem) but has a unique fixed 
point in U. 

Note that since p(f, U) ^ 0, the extension of / to the prime ends compactification 
of U has no fixed point in the circle of prime ends. This implies that / has at 
least one fixed point zq in U, by Brouwer's fixed point theorem applied to eg (U, S). 
Moreover, one can find a closed topological disk D C U whose interior D contains 
all the fixed points of f\u- Let D\ C U be a closed disk containing Dq U /(Dq). 

Since U is homeomorphic to K 2 , we may consider the one-point compactification 
U* = U U {z*} ~ § 2 , in which f\jj induces a homeomorphism /* by fixing z*. Let 
M = [/* \ {zo}, which is homeomorphic to K 2 . Since W = £/* \ D\ is a connected 
neighborhood of z*, and is the unique fixed point of /*\m in W, we see from the 
Extension Theorem |2.3| that there is a map F: M —> M which fixes z* , coincides 
with /* in some neighborhood V of (in M), and has no other fixed point in 
M = U* \ {zq}. Note that since V is a neighborhood of z*, the set K = M \ V is 
a compact subset of U. The map F extends to a homeomorphism F# : U* — > U* by 
fixing zq , and so F*\u is a homeomorphism which coincides with / in U \ K and 
has z as its unique fixed point. In particular defining f'(z) — F*{z) for z £ K and 
f(z) for z € S \ K we have map with the required properties. □ 
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Thus from now on we assume that zq £ U is the unique fixed point of / in U. 
Recall that we are assuming that there is a fixed point in dU. We will consider 
two separate cases. 

Case I. Every connected component of Fix(/) \ {zq} is unbounded. 

In this case, if V is the connected component of (M 2 \ Fix(/)) U {zq} containing 
Zq, then V is a simply connected invariant set whose boundary consists of fixed 
points of /, and U C V. Consider the prime ends compactification cgV . Since 
any accessible point from V of dV is fixed by /, Proposition |5.7| implies that the 
extension / of f\y to cgV has a fixed point in the prime ends circle bf V. Since 
dUC] Fix(/) is not empty (by our assumption), it follows that cl C£ y U n b £ V ^ 0. 
Since f\b £ v is an orientation preserving circle homeomorphism with a fixed point, 
every orbit converges to a fixed point, so the closed, nonempty, /|b £ ^-invariant set 
cl C£ v UDbs V contains a fixed point of /. Thus we may choose Z\ £ cl C£ y UDbg V 
such that f{z\) = Z\. 

By Theorem |4.2| there is N > and a compact set K C U such that there is 
no TV-translation arc contained in W 2 \ K and intersecting d U . Note that / has 
no fixed points in V other than so by increasing the size of K we may further 
assume that K \s & closed topological disk in U containing zq. 




Figure 8. Picture for case I 

Let B\ C V \ K be a cross-section of V such that zq ^ B\ and zi lies in the open 
interval of prime ends of V determined by B±; i.e. Z\ € int C£ y (cl C£ y B\) (see Figure 
[8]). Choose a smaller cross-section B C B\ of V such that ~z\ 6 int C£ y (cl C£ y B) 
and f k {B) C B x for all k with < k < N, and finally let B' C B be a smaller 
cross-section of V such that z\ £ int C£ y (cl C£ y B') and f(B') C S. Note that 
dUDB' 0, since otherwise [/ would contain and that would imply that there 
is some fixed point accessible from U (namely, any point of dV accessible from B') 



contradicting the fact that p{f, U) ^ (and Proposition 5.7). Thus we may choose 
z G dU H B' and an arc 70 C B joining z to f(z). Since there are no fixed points 
in B, we know that 70 does not contain a fixed point, and by Lemma |5.5| there is a 



arc 7 C B for f\y such that 267 (so 7 intersects dU). Since 7 C K. \K, Theorem 



4.2| implies that 7 is not an iV-translation arc, thus we conclude from Lemma 5.6 
(applied to f\y) that {Jh =0 f k ('~f) contains a loop that is homotopically nontrivial 
in V\Fix(/|v) = V\{z Q }. This contradicts the fact that |jf = o f k (l) C which 
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is simply connected and disjoint from zq. This contradiction shows that Case I is 
not possible. 

Case II. There is a bounded connected component of Fix(/) \ {zq}. 

In this case, we may further assume that there is a bounded component of Fix(/) 
intersecting dU. Indeed, let C be a bounded connected component of Fix(/). If 
C intersects dU, there is nothing to do. Otherwise, there is another connected 
component C of Fix(/) that intersects d U (since we are assuming that there is 
a fixed point z\ in dU). We may choose any po G C and consider the space 
M = M 2 U{oo}\{po}j which is homeomorphic to ]R 2 . In this new space, / induces a 
homeomorphism /' for which all our hypotheses hold (the 3-nonwandering condition 
in U and the condition on the prime ends rotation number hold for /' thanks to 



Proposition 3.15 and Corollary 3.9). Since C is a compact subset of R 2 intersecting 
d U and po ^ C", it follows that C is a connected component of Fix(/') intersecting 
8m U, and it is bounded in M (because it is compact). Thus we may replace / by 
/' and have the required assumption. 

Claim 2. We may assume that Fix(/) is totally disconnected. 

Proof. Let V be the connected component of (M 2 \Fix(/))U{zo} containing U, and 
K = R 2 \ V. Clearly V is an /-invariant open set. It follows from the hypothesis 



on the rotation number and Proposition 5.7 that an accessible point from U in 



d U cann ot be fixed by /, so it does not belong to K . Thus d U \ K ^ 0, and by 
Corollary E9] we have that p(f, U, R 2 ) = p(f, U, V). 



From the classification of noncompact surfaces [Ric63j , V is homeomorphic to 
R 2 \E for some totally disconnected closed set E (this can be seen by fixing p € bi V 
and noting that Ci(V) \ {p} is homeomorphic to K 2 and bi(V) \ {p} is totally 
disconnected). Let h: V — > M 2 \ E be a homeomorphism, and consider the map 
hfhr 1 : R 2 \E — > R 2 \E. A standard continuity argument shows that hfh -1 extends 
to a homeomorphism /': M. 2 —> R 2 such that Fix(/') \ {h(zo)} — E. Moreover, 
U' = h{U) is a simply connected open /'-invariant set with a fixed point of /' 
in its boundary, namely the image by h of any bounded connected component of 
Fix(/) that intersects d U (we know that there is at least one such component by 



our assumption in Case II). Again by Corollary 3.9 

p(f, U, R 2 ) = p(f, U, V) - p(f, U', R 2 \E) — p(f, U', R 2 ). 
Therefore /' leaves U' invariant, has a nonzero rotation number in U' , and also /' 



has a fixed point in the boundary of U'. Moreover, the first part of Proposition 3.15 
implies that fv is 9-nonwandering in U , so f'\«?\E is 3-nonwandering in U' . Since 
E is totally disconnected, the second part of the same proposition implies that /' 
is 9-nonwandering in V . Thus the same hypotheses that we have on / hold for /', 
and since Fix(/') is totally disconnected, we may use /' instead of /, proving the 
claim. □ 

Let Z\ £ dU nFix(/). Since the set {z ,^i} C Fix(/) is unlinked (see Remark 



5.3), Theorem 5.4 implies that there exists a closed set X C Fix(/) containing 
zq and z\ such that X is maximal unlinked, and a corresponding isotopy I = 
(f t : R 2 \ X -)• R 2 \ X) te [ ,i] from f = ld R 2\ X to /1 = f\^\x- Moreover, since 
X C Fix(/) is totally disconnected, the isotopy extends to E 2 fixing all points of X; 
thus from now on we assume that 1 = {ft)te[o.i] with f t : K 2 — > R 2 , and ft(x) = x 
for aUte [0,1], x €X. 
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Let X\ = X\{zo}, and let M = M. 2 \Xx- Note that since X is totally disconnected 
and closed, M is connected. Let us denote by Im the restricted isotopy (ft\Ai)te[o.i] ■ 
Let 7? : M — > M be the universal covering of Af and I = (/{)te[o,il the isotopy from 
Idj^ to / = f\ that lifts 1\m- It is easy to see that / commutes with the elements 
of the group of covering transformations Deck(7r). Since the isotopy I\m fixes zq, 
it follows that any element of tt [zq] is fixed by X, and in particular by /. 

Claim 3. If U is a connected component ofTT~ l (U), then U is f -invariant, un- 
bounded and simply connected. Moreover, tt\q is a homeomorphism onto U that 
conjugates f\U to f\jj , f is d-nonwandering in U , and p(f, U) = p(f, U). 

Proof. It is invariant because 7? _1 (U) is /-invariant and U is a connected component 
of that set, which necessarily contains an element of 7r _1 (zo) C Fix(jf). That tt\q 
is injective follows from the fact that U is simply connected. Since d U contains 
z\ € Xx, it follows that U is unbounded. The fact that f\u is conjugated to f\g is 
obvious. 

To prove the last two claims, note that the homeomorphism h = ttq : U —¥ U has 
the special property that it maps cross-cuts to cross-cuts. This implies that / is d- 
nonwandering in U, and it also gives a natural way to extend ft, to a homeomorphism 
between the prime ends compactifications of U and U, which by continuity should 
conjugate the respective extensions of f\jj and f\u- In particular the two extended 
maps have the same rotation number in their prime ends circles. □ 

Claim 4. For any Zx G Xx = X \ {zq}, there is a neighborhood B of Zx in M 2 such 
that 

7r(Fix(/)) n B = 0. 

Proof. We may choose a disk D in M containing zx, disjoint from zq and such that 
the boundary of D is disjoint from X (because X is totally disconnected). Fix a 
neighborhood B of Zx such that for any z in B, the arc (/t(z))te[o,i] is contained 
in D. To show that B has the required property, suppose for contradiction that 
z G B is the image by 7? of a fixed point of /. Then 7 = (ft(z))t£[o,i] is a 1°°P i n 
D, and it must be homotopically trivial in M, because it lifts to a trivial loop in 
the universal covering. Let us show that 7 is homotopically trivial in M \ {zq} as 
well: we know that there is a homotopy (7s) s g[o,i] in M from 7 to z fixing the base 
point z. Let r : M 2 \ D — s- d D be a retract (i.e. r is continuous, r{x) G 3D for all 
x in R 2 \ D and r(x) — x for x G d D), and extend it to M 2 by letting r(x) = x in 
D. Then (r o 7 s ) s e[o.i] is a homotopy from 7 to z which never leaves D. Thus 7 
is homotopically trivial in M \ {zq} — M 2 \ X. But since z G Fix(/) \ X (because 
z G M, z ^ z and M P\ X = {z }), the last part of Theorem 5.4 says that 7 must 
be homotopically nontrivial in M 2 \ X. This contradiction proves the claim. □ 

Fix a connected component U of 7r _1 ((7). 

Claim 5. For any neighborhood V of zx G dU D X, there exists a translation arc 
7 f or f such that 7? (7) C V and 7 intersects d U . 

Proof. Reducing V if necessary, we may assume that V C B, where B is the set 
from Claim |ij so that 7?(Fix(/)) fl V = 0. Since the isotopy I fixes elements of X 
(particularly zx), if z G dU is chosen such that z — 7?(z) is close enough to Zx, one 
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has that 70 = (ft(z))te[o,i\ is contained in V \ X. If 70 = (ft{z))te[o,i], then 70 is 
the lift of 70 with initial point z and its final point is /(z) (recall that / = fx). 
The arc 70 might not be simple, but it necessarily contains a simple arc 71 joining 
2" to f(z), which also satisfies 77(71) C Moreover, 71 con tain s no fixed points 



of /, since 7?(Fix(/)) is disjoint from V. Thus from Lemma 5.5 there is, in any 
neighborhood of 71, a translation arc 7 for / such that z £ 7. In particular, one 
may choose 7 such that ^(7) C V, as required. □ 

To finish the proof, note that from Claim [3] one has that U is /-invariant, simply 
con nected and 9-nonwandering for /, and a — p{f , U) 7^ 0. Thus, from Theorem 
4.2 applied to / and U we obtain a constant N — N ay o and a compact set K C U 



such that no TV-translation arc of / intersecting d U is contained in M \ K. Let 
z\ £ d U n X and consider the set B given by Claim [4] We assume additionally 
that B is simply connected and disjoint from ■k~ 1 {K) (replacing it by a smaller 
neighborhood of z\ if necessary). Note that 7r _1 (£>) C M \ Fix(/), and each of its 
connected components is simply connected. 

Choose a neig hborhood V of z x such that f k (V) C B for < k < N, and 
let 7 C V be the translation arc for / given by Claim [i] Let Tn = (JfcLo/HT)- 
Since ^(r^r) C B, we have that L^r is contained in a connected component of 
7T _1 (_B), which is simply connected and disjoint from Fix(/). In particular, any loop 
contained in L jy is homotopically trivial in M \ Fix(/). Thus, the only possibility 
in Lemma 5.6 applied to /, is that 7 be an TV-translation arc. Since 7 intersects 



4.2 



This 



d U and is contained in M \ K, we get a contradiction from Theorem 
proves that Case II does not hold, completing the proof of the theorem. □ 



5.4. Proof of Corollary 5.2, Suppose there is a fixed point z\ in IR 2 \ U, and U 
is unbounded. Then we consider the new surface M = (M 2 U {00}) \ {zi} which 
is still homeomorphic to M 2 , and the homeomorphism defined from / by setting 



/(oo) = 00. By Proposition 3.15 and Corollary 3.9, the hypotheses of Theorem 5.1 
hold for this new map, so there are no fixed points in the boundary of U in M. 
But this contradicts the fact that 00 is in the boundary of U in M (because U was 
assumed to be unbounded). □ 

6. Generalizations to arbitrary surfaces 

If S is a closed surface of positive genus and U C S is a simply connected 
invariant set with irrational prime ends rotation number for a homeomorphism /, 
we wonder not only whether there can be fixed points in d U but also whether 
U can be too complicated from the point of view of the topology of the ambient 
space. Specifically, can the closure of U be non-contractible in S? Without any 
additional hypothesis, the answer is yes. For example, d U could be a Denjoy-type 
continuum (i.e. the minimal set of the suspension flow on T 2 of a Denjoy example 
in the circle). See Example 9.2 The next theorem, which is Theorem [C| in the 



introduction, shows that if / is 9-nonwandering in U then this is no longer the 
case, and gives additional dynamical information. 

Theorem 6.1. Let f ': S — > S be an orientation preserving homeomorphism of an 
orientable surface S of finite type, and U C S an open f -invariant topological disk 
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such that S\U has more than one point. Assume further that f is d-nonwandering 
in U and p(f, U) is irrational. Then exactly one of the following holds: 

(i) dU is aperiodic, and U is compact and contractible; 

(ii) S is a sphere, and the only periodic point of f in dU is a unique fixed point; 

(iii) S is a plane, and dU is unbounded and aperiodic. 

Moreover, there is a neighborhood W of dU , which can be chosen as an annulus in 
case (i), a disk in case (ii) and the complement of a closed disk in case (Hi), such 
that every connected component of S \dU contained in W is wandering. 

Remark 6.2. For closed surfaces, only cases (i) and (ii) are possible, and in both 
cases d U is contractible. Observe also that the last part of the theorem implies that 
all but one of the connected components of S\ U are wandering in case (i), and all 
connected components of S\ U are wandering in the remaining cases. In particular, 
in cases (ii) and (iii), there are no periodic points in S\U except possibly a unique 
fixed point. 

For the case of surfaces of finite genus (but possibly not of finite type) one can 
easily obtain the following statement, by applying the previous theorem to the ends 
compactification of /. 

Corollary 6.3. If in the hypotheses of the prevoius theorem one replaces the con- 
dition that S be of finite type by the weaker condition that S has finite genus only, 
then one of the following cases holds: 

(i) dU is aperiodic, and exactly one connected component of S\U is nonwan- 
dering (hence invariant) ; 

(ii) S has genus 0, the set of periodic points of f has at most one element (thus 
necessarily fixed), and every connected component of S\U is wandering. 

Finally, in the case that / is nonwandering on a closed surface one has a simpler 
statement, which is Theorem |A| in the introduction. 

Corollary 6.4. Let f : S — > S be a nonwandering homeomorphism of a closed 
surface, and U an open f -invariant simply connected set with irrational prime ends 
rotation number. Then one of the following holds: 

(i) dU is a contractible annular continuum without periodic points; 

(ii) S is a sphere, U is dense in S, and S\U is a cellular continuum containing 
a unique fixed point and no other periodic points. 

Proof. If the first case in Theorem |6.1| holds, then d U is an aperiodic invariant 
continuum, and so by Theorem |2.4| it is an annular continuum as well, so the first 
case of the corollary holds. On the other hand, if the second case of Theorem |6.1| 
holds, then since / is nonwandering and all components of S \ U are wandering, 
it follows that S \ U — 0, so that dU — S \ U is nonseparating (i.e. a cellular 
continuum) . □ 

Remark 6.5. Examples |9.6| and |9.5| show that the case with a fixed point cannot be 
excluded, even in the smooth area-preserving setting. 

6.1. Proof of Theorem |6.1| , We begin considering closed surfaces only. 

Claim 1. If S is a closed surface, then one of cases (i) or (ii) holds. 



:50 



ANDRES KOROPECKI, PATRICE LE CALVEZ, AND MEYSAM NASSIRI 



Proof. We consider first the case where S is a sphere. Then S\U is a non-separating 
/-invariant continuum. We claim that there is a fixed point z € S \ U . This is a 
consequence of Cartwright-Littlewood's theorem, but alternatively one can prove it 
as follows: Since p(/, U) ^ 0, if D C U is a closed topological disk such that <9.D is a 
loop disjoint from Fix(/) and homotopic to the circle bf (£/, 5) in Cf(C/, 5) \ Fix(/), 
then the fixed point index of / in U is i(f,D) = 1. Since / preserves orientation 
and S is a sphere, Lefschetz' formula implies that i(f, D) + i(f, S \ D) = 2, and in 
particular there is a fixed point in S \ D. But there are no fixed points in U \ D, 
so there must be a fixed point in S \ U as claimed. 

Let S 1 = S \ {zq} ~ M 2 . Then ds' U has more than one point, and by Corollary 



3-9 p(f\s',U,S') = p{f,U,S). Also, f\s> is 9-nonwandcring in U by Proposition 



3.15 Hence by Theorem 5.1 applied to f n \s' for each neNwe conclude that there 
is no periodic point in $5/ U. 

Suppose first that zo ^ ds U. Then d$U = ds> U, and so there are no periodic 
points in ds U. This means that ds U is aperiodic. Since zq G int S \ U, it follows 
that U 7^ S, and since S is a sphere this easily implies that U is contractible, so 
case (i) holds. 



Now suppose that zq € ds U. Then U is unbounded in 5', and Corollary 5.2 
implies that there is no periodic point in S" \ U. Thus the unique periodic point of 
/ in S\ U is Zq. This proves that case (ii) holds, completing the proof of the claim 
in the case that S is a sphere. 

Now assume that g is a surface of genus g > 1. The case g = 1 is similar, and 
is explained at the end of the proof. Let 7? : D — > S be the universal covering of 
S. We may assume that D is endowed with the hyperbolic metric and the group of 
covering transformations Deck(7r) consists of isometrics of D. There is some lift / 
of / that fixes a connected component of U of 7r _1 (C/). Claim [3] from the proof of 
Theorem |5 . 1| remains valid in the present setting, with the same proof, so w e have 



that U is 3-nonwandering for / and p(f, U) = U). From Theorem 5.1 applied 
to /" for each n E N, we conclude that there are no periodic points of / in dU. 
Let us divide the remainder of the proof of Claim [l] into two sub-claims. 

Claim 1.1. clpf/nS 1 =0 (i.e. U is bounded inD). 

Proof. By the theory of Nielsen, the map / extends to a the boundary circle S 1 , 
defining a homeomorphism F from D = D U S 1 to itself. Moreover, there is n G N 
such that /" has a fixed point in dD (this is for instance the content of |Nie32[ 
Theorem lQ. We may extend F to the whole plane C continuously, for instance 
by letting F{re u ) = rF{e lt ) for r > 1, t € R. 

Therefore, the map F n : C — > C leaves invariant the open 9-nonwandering topo- 
logical disk U CD and also the disk D, and F n has a fixed point in S . Since F n \§i 
is a circle homeomorphism with a fixed point, the orbit of any point in S 1 converges 
to a fixed point of F"|gi. 

Assume for contradiction that the claim does not hold, i.e. <9p U n S 1 7^ 0. Since 
the latter is a closed F"-invariant set, it follows from the previous obs ervat ion that 
<9jj U contains a fixed point of F n (which lies in S 1 ). From Corollary 



3.9 



we have 

that p(F n , U, C) = p(f n , U, D) which is just np(f, U, D), which in turn is equal to 
np(/, U), an irrational rotation number. 



^An english translation of INie32l can be found in |Nic86 
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Let us show that F is 3-nonwandering in U (and hence so is F n for any n £ N). 
Since we already know that F\q — f is 9-nonwandering in U, it suffices to show 
that any wandering cross-section of U in C contains some cross-section of U in D. 
Let D be any cross-section of U in C so that 7 = djj D is a cross-cut of U in C. 
Observe that 9c -D C clc 7 U S 1 U do U. Moreover, if dc D \ clc 7 intersects <9n 
at some point z £ D, then we may easily obtain a cross-section D' of [/ in D which 
is contained in D, by an argument already used in Section [3} letting C be a small 
enough circle around z (so that it bounds a disk disjoint from clc 7) , one has that 
any connected component of D n C is a cross-cut of U contained in D, and one of 
its cross-sections must be contained in D, as required. 

Thus we need to show that dc D \ clc 7 always intersects do U. If this is not 
the case, then dc D \ clc 7 C S 1 , so that D is bounded by clc 7 U S 1 . This implies 
that both endpoints of clc 7 are in S 1 and they are different (otherwise D would 
not be a cross-section of U in C). Thus D is a cross-section of D. But from the 
fact that S is a closed surface it follows that given z G S, any cross-section of D 
contains infinitely many elements of 7r -1 (z), contradicting the fact that tt(U) is 
simply connected. 

We have thus obtained an orientation-preserving homeomorphism F n of the 
plane with a 9-nonwandering invariant open topological disk U which h as a fixed 
point in the boundary, and p(F n ,U,C) 7^ 0. This contradicts Theorem 



5.1 



com- 
pleting the proof of Claim [O] □ 

Thus the set K = cl^U is a compact subset of D. Let K be the "filling" of K , 
i.e. the union of Kg with all the bounded (in D) connected components of D \ Kq. 

Claim 1.2. tt\k is injective. In other words, K n T(K) — for any Deck trans- 
formation T 7^ Id. 

Proof. Consider the family of sets H = {T(K) : T <E Deck(Tr), T(K) flJC^}. 
The elements of H are permuted by /. Moreover, since Deck(7r) acts properly on 
D, there are finitely many elements in H. Therefore, there is k G N such that / 
fixes every element of H . Assume for contradiction that H contains some element 
T{K) with T ^ Id. Since U is disjoint from T(U), we have that U D dT(U) = 
as well. Since dT(K) C dT(U), we deduce that U (1 dT(K) = 0. The fact that 
Deck(7r) acts properly on D also implies that T _1 (i^) <f_ K, so that K <f_ T{K). 
This implies that dK intersects D\T(K), so that U £ T{K) (because dK C d U). 
The latter, together with the fact that U is disjoint from dT(K), implies that U 
is disjoint from T(K). Observe that since T(K) intersects both K and D \ K, we 
have that dK C dU intersects T(K). 

Since K is a non-separating continuum, D \ T(K) is homeomorphic a one- 
punctured plane K 2 \ {zo}. The map g induced by in E 2 \ {z } extends to 
M. 2 by fixing zq (because fixes T(K)), and the set U corresponding to U after 
this identification is a 9-nonwandering invariant topological disk for g in K 2 due to 
Proposition |3.15| Moreover, by Corollary |3.9| 

p(g, U , M 2 ) = P (f k | DVr(J0 ,U,B\ T{K)) = p(f k ,U,B) = np(f, U, D) f 0. 
Thus Theorem 



5.1 



implies that g has no fixed points in d U, contradicting the fact 
that z is in the boundary of U (which follows from the fact that d U intersects 



T(K)). This contradiction proves Claim 1.2 □ 
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Since K is compact and connected, and we showed that it projects injectively 
to S, it follows that ir(K) is contractible in S. Since U C n(K), we conclude 
tha t U is contrac tible in S as well. Finally, as we already explained before Claim 
1.1 by Theorem 5.1 there are no periodic points of / in dU, and since 7r| cl ^ is 
a homeomorphism onto U and nf — fir, we conclude that there are no periodic 
points of / in d U as well. This completes the proof of Claim [l] when S has genus 
<?>!• 

It remains to consider the case where the genus of S is g — 1, that is, when 
S ~ T 2 . But in this case, the same proof used for the case g > 1 works, after 
observing the following facts: any lift / : M 2 — > M 2 of a homeomorphism / : T 2 —> T 2 
has the form f(x) = Ax + 4>{x) where A 6 GL(2,Z) and is a Z 2 -periodic map. 
This implies that if one compactifies M 2 with a "circle at infinity" Soo ~ S 1 with 
the topology where a neighborhood system of re 1 *, for r > 0, is given by the open 
sets V €i m = {re ts where r > M and \s — t\ < e}. Since / = A + (j>, it follows easily 
that the map / extends to M 2 U ~ D continuously, and this extension depends 
only on A (in fact, if v = re lt , then the extension maps v to Avj ||Au||). 

Moreover, since A £ GL(2,Z), one has from classical results that some power 
of A has a real eigenvector. Therefore there is n > and v £ Soo such that 
A n vj \A n v\ = v, and so the extension of /" to M 2 U ~ D has a fixed point in 
the boundary circle. Thus in Claim [Ll] we may use these facts instead of the result 
of Nielsen, and the rest of the proof works without any modification for this case. 
This completes the proof of Claim [T] □ 

Claim 2. If S is closed, then there a neighborhood W of dU, which is an annulus 
in case (i) and a disk in case (ii), such that every connected component of S\dU 
contained in W is wandering. 

Proof. ByClaim[lJ one of cases (i) or (ii) holds. Assume first that case (ii) holds, so 
S is a sphere, and there is a fixed point z\ € d U . We will show that every connected 
component of S\U is wandering, which implies that the disk W = S\{zq] satisfies 
the claim for any zq £ U . Suppose on the contrary that there is a nonwandering 
component V of S \ U. Since U is invariant, it follows that f n (V) = V for some 
n. Since U is connected and S is a sphere, V is simply connected. Choose xq £ V 
and let h : S S be a homeomorphism h: S —¥ S which is the identity outside 
a compact subset of V and such that h(f n (xo)) — xq. Then /' = hf n is a new 
map which coincides with / in a neighborhood of U and f' n (xo) = x . But this 
contradicts Theorem 



5.1 



applied to / /n |s\{a; }. This proves the claim when case (ii) 
holds. _ 

Now suppose that case (i) holds, so U is contractible. This means that there is 
a closed topological disk B C S that contains U in its interior. Choose any point 
zq £ U and let W = B \ {zq}. To prove that W satisfies the claim, assume for 
contradiction that there is some nonwandering component V± of S \ U such that 
V\ C W . Then V\ is periodic, and we may assume f{V\) = V\ by considering a 
power of / instead of / (while keeping the remaining hypotheses). 

Let us show that it suffices to consider the case where S is a sphere. If S is not 
a sphere, then its universal covering space S is homeomorphic to M 2 . If ir: S — > S 
is the covering projection, there is a connected component U of 7r -1 (£/) and a lift 
/: S — > S of / such that f(U) — U. Let B be the connected component of 7r _1 (i?) 
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containing U, so that 7r|g is a homeomorphism onto B, and let V\ be the connected 
component of 7r _1 (Vi) in B, so that f(V\) = V\. Let S' = S U {00} be the one- 
point compactification of S (so S' is a sphere), /' the map induced on S' by /, and 
U' = U. The facts that tt/' = /V holds on clg J7 and 7r | cl _ q is a homeomorphism 

onto U imply that /' is 9-nonwandering in U' and p(f, U') is irrational. Moreover, 
if V[ = V\ and V 2 ' is the connected component of S' \ clg U containing 00, then V[ 
and V 2 ' are both /'-invariant connected components of S' \ clg/ [/'. Thus we are in 
the same setting as in the beginning of the proof, but on the sphere. 

Thus we assume that S is already the sphere and we will find a contradiction. 
Choose any point Zi £ Vi and let Bi be a closed topological disk containing Zi and 
f(zi). If 7^ : [0,1] — > d Bi is a parametrization of d Bi (oriented positively), then 
ji is nomotopic to / o 7^ in Vi \ {f(zi)} (because / preserves orientation), and so 
by Theorem 2.2 applied to Vi \ {f(zi)} there is a homeomorphism hi : Vi \ {f(zi)} 
such that /ij o / o 7, = 7^ (note that this means that hif fixes each point of d Bi), 
and hi is fixed outside a compact subset of Vi \ {f(zi)}. 

We may define h: S -> S by h(z) = h t (z) if z G V$, and = « if z £ V\ U Vi, 
which is a homeomorphism of S. Then /' = hf coincides with / on U (in particular 
p(U, f) is irrational and /' is 9-nonwandering in U) and has a pointwise fixed circle 
d Bi inVi fori £ {1,2}. Consider the new surface S' = S'\(inti?iUinti?2)/^, where 
~ is the relation that identifies dB\ with d B2 by Jt(t) ~ 72(1 — t) for x £ S 1 (the 
orientation of 72 was reversed so that the identification gives an orientable surface 
again, see Figure [9]). Then S' is a torus, and after the identification, V\ \ int(-Bi) U 
V2\hit(-B2) becomes a topological annulus A in S' which is a connected component 
of S' \ U. 




Figure 9. Identifying the boundaries of B\ and B2 to obtain a torus 



Let g' : S 1 — > S' be the map induced by /'. Observe that since dU is aperiodic 
for /, it is aperiodic for g' as well. Since g' is 9-nonwandering in U and p{g' , U) is 
irrational, Claim [l] applied to g' and S' implies that dU is contractible in S' . But 
that means that dU is contained in some closed topological disk B, and so one of 
the connected components of S' \ U is a punctured torus and the remaining ones are 
open topological disks. But as we already mentioned, A is a connected component 
of S' \ U homcomorphic to an annulus; so we arrived to a contradiction, proving 
the claim. □ 
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Claims 1 and 2 complete the proof of the theorem in the case that S is closed. 
To finish the proof we need to consider the case where S is a non-closed surface 
of finite type. In this case, the ideal boundary bi(5) is finite, so if S' = ci(S) 
and /': S' — > S' is the extension of /, then bi(S') is a finite /'-invariant set, which 
must therefore consist of finitely many periodic orbits. Again by Corollary |3.9| 



and Proposition 3.15 the hypotheses of the theorem remain valid for U and /'; in 
particular since S' is a closed surface, what we already proved implies that one of 
the cases (i) or (ii) holds for /'. 

Suppose first that case (i) holds for 5" and /'. Then ds> U is aperiodic and els' U 
is contractible in S'. Since points of bj(S) are periodic for /', it follows that ds> U 
is disjoint from bi(5). Note that this implies that dsU — ds' U, so dU — dsU 
is aperiodic. Also we have an annular neighborhood W containing d U such that 
every connected component of S' \ dg> U contained in W is wandering. One of the 
boundary components of d W must be contained in U, so B — U U W is an open 
topological disk containing U. Since points of bi(5) are disjoint from dU and from 
any connected component of S' \ d U contained in J5, this means that the finite set 
bi(S') is contained in the unique connected component V of S' \ dU not contained 
in B. It follows that there is a smaller open topological disk B' C B such that 
U C B' and B' is disjoint from bi(S'). Thus B' C S, and U is contractible in S. 
Thus case (i) holds for S and /. Also, letting W = B'\ {zo} for some point zq e U, 
we obtain an annular neighborhood of d U with the required property, completing 
the proof in this case. 

Now assume that case (ii) holds for S' and /'. Then S' is a sphere, and every 
component of S" \ cls> U is wandering for /'. Moreover, ds> U has a unique fixed 
point zq and no other periodic point, and there is an open topological disk W' 
containing ds> U such that every connected component of S' \ U contained in W' 
is wandering. Since U is invariant, S'\ W intersects U, so dW C U. This implies 
that S' \ cl s > U C W . 

Consider first the case where Zq € bi(5). Then, since bi(5) consists of periodic 
points of /' and there are no other periodic points in ds> U, it follows that bi(5) \ 
{zo} C S'\c\s' U . But since all connected components of S'\c\s'{U) are contained 
in W , they are wandering, so they cannot contain a point of bi(S'). This implies 
that bi(5) = {zq}, hence S is homeomorphic to M 2 and dU — ds U is aperiodic. 
Thus, case (hi) holds for / and S. Moreover S'\W is a closed subset of U, so taking 
any closed topological disk B C U containing S' \ W we have that W = S \ B 
has the property that every connected component of S \ d U in W is wandering, as 
required. 

Finally, consider the case where zq ^ bi(S'). Then zq £ ds U, and ds> U \ {zq} 
contains no periodic points of /'. This implies that bi(S) C S' \ els' U. But every 
component of 5"\cls' U is wandering (because it is contained in W), so we conclude 
that bi(5) = 0. This means that S — S' and /' = /, and the theorem holds. This 



completes the proof of Theorem 6.1 □ 



7. Generalizations to non-simply connected sets 

In the case that U is not simply connected, we can adapt our previous results 
to the next somewhat technical, but potentially useful result. Recall the notation 
and definitions from Section [2] |3.3| and j ]3.4| 
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Theorem 7.1. Let f: S — > S be an orientation-preserving homeomorphism of a 
closed surface S , and U C S an open f -invariant connected set. Let p be a regular 
fixed end of U . If f is d-nonwandering at p, and p(/, p) is irrational, then one of 
the following holds: 

(i) Z{p) is aperiodic, or 

(ii) Z{p) is contractible and has a unique fixed point and no other periodic point. 
Furthemore, there is a neighborhood W of Z{p), which is an annulus in case (i) 
and a disk in case (ii), such that every connected component of S\ Z(p) contained 
in W is wandering. 

Again, in the nonwandering setting we have a simpler statement: 

Corollary 7.2. Let f ': S — > S be an orientation-preserving nonwandering home- 
omorphism of a closed surface S, and U C S an open f -invariant connected set. 
Let p be a regular fixed end of U . If p(f,p) is irrational, then one of the following 
holds: 

• Z{p) is an aperiodic annular continuum, or 

• Z{p) is a cellular continuum with a unique fixed point and no other periodic 
points. 



A version of Theorem 4.2 can also be stated in this setting: 



Theorem 7.3. Let f : S — > S be an orientation-preserving homeomorphism of an 
orientable surface S of genus g < oo, and U C S an open f -invariant connected 
set. Let p be a regular fixed end of U such that f is d-nonwandering at p, and 
p(f, p)=a^0. Then there is a constant N a ^ g 6 N and a p-collar C such that any 
N a g-translation arc that intersects Z(jp) also intersects U \ C. 



7.1. Proof of Theorem 7.1 The main idea to obtain theorem 7.1 from the pre- 
vious results is to use surgery to reduce the problem to one in which U is simply 
connected. Assuming the hypotheses of the theorem, consider the ends compacti- 
fication Ci(U) and the extension /* of f\jj to Ci(J7). Since p is a regular end, p is 
an isolated fixed point of /* . Let 70 be a simple loop bounding a p-collai[^] Dq in 
U, and 7 another simple loop bounding a p-collar D such that D U f{D) c -Do- 
Then 7' := 7(7) is homotopic to 7 in Do- Let D\ be another p-collar such that 
D\ C D n f(D). Then 7 and 7' are homotopic simple essential loops in the annulus 



A = int£>o \ Di, and it follows from Theorem 2.2 that there exists an orienta- 
tion preserving homeomorphism h: A — >• A such that ^1(7') = 7 and h(x) = x 
for 1 £ 94. We may extend h as the identity in S\ A, and define /' = hf, so 
that f'{"j) = 7 and /' coincides with / outside A. In particular, the fact that 
/' coincides with / in the p-collar Di implies that /' is d-nonwandering at p and 
p(f,P) = P{f,P) is irrational. 

Let us assume that 7 is non-separating in S (the remaining case is similar and 
explained at the end of the proof). Let So = S \ 7 (for the separating case, one 
would choose the connected component of S \ 7 containing D\). Let S' = c\Sq, 
and denote by g' : S' — > S' the extension of f'\s to S'. If Ai denotes the connected 
component of A \ 7 bounded by 7^, for i £ {0, 1}, then each A, is a collar of one 
of a different end e» € bi(So), an d bi(So) = {^o, So the sets A\ = Ai U {e^} 



2 recall from f 3.3 



this means that D is a subset of U bounded by a simple loop such that 



D U {p} is a closed topological disk in Ci(£/) 
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are topological disks which are neighborhoods of e,, for i £ {0, 1}. Observe that g' 
coincides with / on S' \ (A' U A[). 




Figure 10. U' and the ends compactification of So 

Since the connected component of Sq \ Z(p) containing D\ is precisely int_D, 
which is /'-invariant and contains a unique end of So (namely, ei), it follows that the 
connected component of S'\Z(p) containing D\ is a topological disk U' = DU{ei} 
such that ds> U' = Z(p) (the latter being true because D\ is a p-collar). Clearly 
p(g', U') = p(f,p) and U' is 9-nonwandering for g'. We now apply Theorem |6.1| to 
g' and U' , and translate the information to / and U. 

Suppose first that case (i) in Theorem |6.1| holds for g' and U' . Then $5/ U' = 
Z(p) is aperiodic (so case (i) holds for / and U), and there is an annular neighbor- 
hood W of Z(p) such that any connected component of S' \ Z(p) contained in W 
is wandering. Note that the connected components of 5" \ Z(p) containing ei and 
t2 cannot be contained in W' , so by reducing W' we may assume that it does not 
contain ei or ti. Thus W is also an annular neighborhood of Z(p) in Sq (hence 
in S). Since W does not contain U, if V is a connected component of S \ Z(p) 
contained in W' , then V is disjoint from U . Thus V is also a connected component 
of S' \ Z(p), and therefore V is wandering for /'. Since / = g' in S \ U, it follows 
that V is wandering for /, so the final claim of the theorem holds with W = W . 



Now suppose that case (ii) from Theorem 6.1 holds for U' and g' . Then 5' 
is a sphere, and there is exactly one fixed point in ds' U' = Z(p). Moreover, all 
connected components of S' \ cls> U' are wandering. This is not possible in this 
case, since the connected component of S' \ c\s' U' containing eo is invariant; thus 
case (ii) cannot hold. 

Finally, we need to consider the case where 7 is separating in S. We sketch the 
proof since it is similar to the previous case. For this case, we choose Sq as the 
connected component oi S\j that contains D\ (see Figure |TT]). In this case, Sq has 
a unique end ei, and if S' = Ct(5o) = <SoU{ei}, then the connected component U' of 
So that contains ei is a topological disk with p(g', U') = p(f,p) and ds> U' = Z(p) 



(where g' is defined as before). Applying Theorem 6.1 again for g' and U 1 leaves us 
with two cases; if case (i) holds for g' , then virtually the same argument as before 
shows that case (i) holds for /. On the other hand, if case (ii) holds for g' then 
one has that S' is a sphere and all connected components of S' \ U are wandering. 
This implies that S' \ {ci} = So is a topological disk containing Z(p), so Z(p) is 
contractible in S, and all connected components of S \ Z(p) contained in So are 
wandering; thus case (ii) and the final claim of the theorem hold with W = Sq. 
This completes the proof of the theorem. □ 
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Figure 11. Case with 7 separating S 



7.2. Proof of Corollary |7.2[ The nonwandering condition implies that / is d- 



nonwandering at p. If the first case from Theorem 7.1 holds, then Z(p) is an 
aperiodic invariant continuum contained in an annulus V , and all connected com- 
ponents of S \ Z(p) contained in V are wandering. Since / is nonwandering, it 
follows that no connected component of S \ Z(p) is contained in V. This easily 
implies that Z(p) separates V into exactly two annular connected components, so 



it is annular (alternatively, Theorem 2.4 implies directly that Z(p) is annular) 



If the second case from 7.1 holds, then Z(p) has a unique fixed point, no other 
periodic point, and has a neighborhod D homeomorphic to a disk such that every 
component of S \ Z(p) contained in D is wandering. This means that D \ Z(p) 
has no relatively compact (in D) connected components, and since Z(p) is compact 
we conclude that Z(p) is non-separating in S. This implies that Z(p) is cellular, 
completing the proof. □ 



7.3. Proof of Theorem 7.3 , We only sketch the proof since it is very similar to 
the proof of Theorem |7.1[ The surgery from the proof of Theorem |7.1| allows us to 
obtain a simply connected set U' on a surface <S" = Ci(Sq), where Sq is a connected 
component of S \ 7 for some loop 7 bounding a p-collar. Moreover, / coincides 
with <?' outside an annular neighborhood A C U of 7, and p(g' , U') = p(f,p) = a. 
Note that the genus of S' is not greater than the genus of S; thus letting TV — 



maxo<fc< 9 N a ^ (Where N a ^ is the number from Theorem 4.2) we have that there 
is a compact set K C U' such that any TV-translation arc for g' intersecting dU 
must also intersect K. Let C be a p-collar disjoint from A such that K c U \ C , 
and let C be a smaller p-collar so that f n (C) C C for < n < N. Suppose for 
contradiction that there is an TV-translation arc 7 for / such that 7 fl (U \ C) = 0. 
Then f k (j) is disjoint from U \ C for < k < N, and since U \ C C S \ A, this 
implies that 7 is also an TV-translation arc for g' intersecting d U'. But 7 is disjoint 



from U'\C , and thus disjoint from K, contradicting Theorem 4.2 due to our choice 



of TV. □ 



8. Applications to generic area-preserving diffeomorphisms 

The main theorem of this section is an application of our previous theorems that 
generalizes Mather's results from [M at81j in two ways: it works for any r > 1 (since 
it does not rely on Moser stability of elliptic points), and the conclusion is much 
stronger (since one gets that there is no periodic point, instead of just saying that 
the prime end rotation number is irrational). Before stating the theorem, we need 
a definition. 
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Given a closed surface S, denote by Q r the set of all area preserving C r diffco- 
morphisms of S satisfying the following two properties: 

(Gl) All periodic points of / are either hyperbolic or elliptic, and there are no 
saddle connections; 

(G2) If p is an elliptic periodic point and U is a neighborhood of p, then there 
is an open disk D containing p, contained in U, and bounded by finitely 
many pieces of stable and unstable manifolds of some hyperbolic periodic 
orbit q intersecting transversely (see Figure 12 1. 




Remark 8.1. Robinson proved that, for any r > 1, property (Gl) is C r -generic 
|Rob70| . and (G2) is C r -generic due to Zehnder |Zeh73j . Thus Q r is residual in the 
space of area preserving C r diffeomorphisms of S. 

Remark 8.2. The idea of replacing the Moser genericity condition from Mather's 
arguments by Condition (G2), thus removing the requirement that r be large, 
was also used by Girard [G"ir09] to prove that C r generically (r > 1) for an area 
preserving diffeomorphism there is no invaraint curve of rational rotation number. 

Theorem 8.3. // / is a C r -generic area preserving diffeomorphism of a closed 
orientable surface S (namely if f € Q r , r > 1), and p is a regular periodic end of 
an open f -invariant connected set U , then Z(p) is an aperiodic annular continuum. 

Remark 8.4. For the case of the sphere, this theorem can be proved using a classic 
result of Pixton [Pix82j (combined with results of Mather |Mat81j ) which guarantees 
that under the generic hypotheses of the theorem, every hyperbolic periodic point 
has a homoclinic intersection. In fact, this is done in |FLC03| . A similar argument 
can be done on the torus using a result of Oliveira |01i87j . However, for surfaces of 
higher genus, is it not known whether a C r -generic diffeomorphism has a homoclinic 
intersection for each (or even for some) hyperbolic periodic point. Furthermore, for 



maps in Q r (as in Theorem 8.3) it is actually not true that some hyperbolic periodic 
point has a homoclinic intersection: one may produce an example in the surface of 
genus 2 which is the time-one map of a flow with exactly two hyperbolic saddles 
and no saddle connection or additional periodic points (where every orbit other 
than the saddles is dense). 
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8.1. Saddle hyperbolic fixed points. We begin with a result about hyperbolic 
points only which is a consequence of Theorem |4.2| 

If p is a hyperbolic fixed point for a surface diffeomorphism, a branch of p is a 
connected component of W s (p) \ p or W u (p) \ p. There are two stable branches 
and two unstable ones. If / maps each branch to itself, then we say / is branch- 
preserving at p. 

Theorem 8.5. Suppose that f:S^>-Sisa homeomorphism of a closed ori- 
entable surface, and U C S is an open f -invariant topological disk on which f 
is d-nonwandering and p(f, U) is nonzero. Assume further that f is C 1 in a neigh- 
borhood of dU. Then there is no branch-preserving hyperbolic fixed point in dU. 

Proof. Suppose p € dU is a branch-preserving hyperbolic fixed point. Using a 
local linearizing neighobhrood near p, one sees that for each N E N there is a 
neighborhood Vn of p such that Vn \ {p} is foliated by TV-translation arcs, so by 
Theorem |4. 2 1 we have that Viv does not meet dU if N is large enough, contradicting 
the fact that p&dU. □ 

8.2. Proof of Theorem |8.3[ First, we show that the prime ends rotation number 
p(f,p) is irrational. This, in fact, is a theorem of Mather [Mat81j Theorem 5.1] 
with the slightly different assumptions. While Mather assumes the Moser stability 
of elliptic points, we instead assume Property (G2). This changes only the part 
of Mather's proof in which it is shown that there are no elliptic periodic points in 
Z(P). 

To see that Z(p) contains no elliptic periodic point, suppose for contradiction 
that p G Z{p) is an elliptic periodic point. Since Z(p) consists of more than a 
single point, we may choose a neighborhood V of p that does not contain Z(p). By 
Property (G2), there is some hyperbolic periodic point q and a disk D bounded 
by finitely many arcs of the stable and unstable manifolds of q such that p € D 
and D C V. We may write 3D = (JILi W» where each Wi is an arc of stable or 
unstable manifold of an iterate of p, and we order them cyclically, i.e. in a way that 
Wi intersects Wi+i for 1 < i < n — 1. 

By Mat81, Corollary 8.3], if a branch of the stable or unstable manifold of q 
intersects Z(p), then the whole branch is contained in Z(p). Since Z(p) is connected 
and contains a point in the complement of D, it follows that d D intersects Z(p). 
Thus Wi intersects Z(p) for some i. Assume without loss of generality i = 1. Then 
as we mentioned [Mat81| Corollary 8.3] implies that W\ C Z(p). Since Wi intersects 
W\ C Z(p), repeating this argument we have that Wi C Z(p), and by induction we 
conclude that d D = 1J™ =1 Wi C Z(p). But U is connected and intersects both D 
and S \ D (because Z(p) is not entirely contained in V and D C V). This means 
that U intersecs dD a Z(p), contradicting the fact that Z(p) C dU. Thus there 
are no elliptic periodic points in Z(p), and p(f,p) is irrational. 

The key part of the proof is Corollary |7.2| which now implies that either Z(p) 
is an aperiodic annular continua, which is what we wanted to show, or Z{p) is 
a cellular continuum with empty interior and with a fixed point p. Assume the 
latter case. Since we showed that p cannot be elliptic, condition (Gl) implies that 
p is hyperbolic. The fact that Z(p) is contractible implies that there is an open 
topological disk D C S such that 3D is a loop in U and Z(p) C D. Suppose that S 
is not a sphere, so its universal covering S is homeomorphic to K 2 . If ir : S — > S is 
the universal covering map, p G n^ 1 (p) and / is a lift of / such that f{p) — p, then 
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there is a disk D C S such that tt\^ is a diffeomorphism onto a neighborhood of 
Z(p). Let K be the connected component of ■n^ 1 (Z(p)) in D. Then if is /-invariant 
because it contains a fixed point p, and if has empty interior. Let S' = SU {00} be 
the one-point compactification of S, which is a sphere, and /' is the map induced 
by /. Then U' = S' \ K is a topological disk, d U' = K, and /' is C 1 in the 
neighborhood D of K. The fact that 7rg/ = f'%g easily implies that /' is d- 
nonwandering and p(f, U') = p(f, p). Since /' has a hyperbolic fixed point p in U', 
which we may assume branch-preserving by using f' 2 instead of /, this contradicts 
Theorem 18.51 



Thus there are no periodic points in Z(p), and it follows from Theorem 2.4 that 



Z{p) is an annular continuum, completing the proof. □ 



Remark 8.6. After using Corollary 7.2 which guarantees that Z(p) is contractiblc 
(if it contains a fixed point), it would also be possible to finish the proof using 
arguments such as Pixton's theorem Pix82] to conclude that there is a homoclinic 
intersection of p, from which one easily obtains a contradiction (see also [KNllj ). 
Note that this proof would not work without the aid of Corollary 7.2 since an 
analogous of Pixton's theorem is not available on surfaces of genus greater than 1. 

8.3. Aperiodicity of the boundary of complementary domains. Recall that 
a complementary domain is any connected component of the complement of a 
nontrivial continuum. A complementary domain always has finitely many ideal 
boundary points (see, for instance, [Mat81[ Lemma 2.3]), and none of its boundary 
components are single points. Therefore, Theorem [E] from the introduction follows 
immediately from the next result. 

Corollary 8.7. Let f be a C" -generic area preserving diffeomorphism of a closed 
orientable surface S (namely, let f £ Q r , r > 1), and U a periodic open set with 
finitely many ends. Then the boundary of U is a finite disjoint union of aperiodic 
annular continua and periodic orbits. 

Proof. Since bi(t/) consists of finitely many points which are periodic by the map 
induced by / on Ci(f7), one may choose n € N such that /" fixes every ideal 



boundary point of U, and it follows from Theorem 8.3 applied to f n that for each 
p € bi(U), if Z(p) has more than one point then it is an aperiodic annular continuum 
invariant by Since there are finitely many ends, is easy to see from the definition 
of ideal boundary that d U = U p ebi(;y) %(P)- Thus d U is a finite union of aperiodic 
(but possibly non-disjoint) /"-invariant annular continua, together with a finite set 
of fixed points of /" (corresponding to the ends where Z(p) is a sinlg epoint). In 
particular, d U has finitely many connected components, and the ones that are not 
periodic points are aperiodic /"-invariant continua. By Theorem \2A\ this implies 
that each connected component of d U that is not a periodic point is in fact an 
aperiodic annular continuum, completing the proof. □ 

Remark 8.8. This result (with a minor variation in the generic conditions) is mis- 
takenly attributed to Mather in |KN10( Theorem 12], leaving a gap in the article. 
In [KN11 , this mistake is corrected by avoiding the use of Theorem 12 from [KN10J. 



Using Corollary 8.7 there is no gap and the erratum |KN11] is no longer necessary. 

8.4. Closure of branches of stable and unstable manifolds. The following 
results are proved in |Mat81) under different assumptions (which are C r generic, 
only for large r). 
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Corollary 8.9. Let f be a C r -generic area preserving diffeomorphism of a closed 
surface S (namely, let f £ Q r , r > 1). Suppose p is a hyperbolic periodic point. 
Then, 

(1) If p belongs to some periodic continuum K , then then the stable and unstable 
manifolds of p are contained in K . 

(2) All four stable /unstable branches of p have the same closure. 

Proof. To prove (1), let 7 be a branch of p. Suppose for contradiction that 7 <£. K . 
Then, 7 intersects a connected component V of S \ K . This implies that p £ d V. 
But V is a periodic complementary, so this contradicts Corollary |8.7[ 

To prove (2), let 71 and 72 be two branches of p. Then cl(7i) is a periodic 
continuum and p £ K\, so it follows from (1) that 72 C cl(7i). Similarly, 7! C 
01(72), completing the proof. □ 

8.5. Density of stable manifolds. In [FLC03] it is proved that for a C r generic 
area-preserving diffeomorphism of the sphere, the union of stable manifolds of hy- 
perbolic period points is dense. One of the reasons for the interest in this type of 
results is that they aim in the direction of one of the oldest open problems in dy- 
namics, i.e. the conjecture of Poincare on the density of periodic points for generic 
area preserving diffeomorphisms. 

Corollary |8.7| fills a gap in the proof of the generic density of stable manifolds 
for arbitrary surfaces given by Xia in |Xia06j , (particularly, in the proof of Lemma 



2.3) which is similar to the one mentioned in Remark 8.8 Thus, we can state that 
the following theorem holds. 

Theorem 8.10. For a C r generic area-preserving diffeomorphism of a closed sur- 
face, the union of stable manifolds of hyperbolic periodic points is dense. 

The second claim from the main theorem of |Xia06j states that an open set 
without periodic points is necessarily contained in the closure of the stable manifold 
of a given periodic point. This seems to be more delicate, and we are unable to 
complete its proof. 

9. Examples 

9.1. Necessity of the 9-nonwandering hypothesis. 

Example 9.1 (Many fixed points). The 9-nonwandering hypothesis assumed in 
Theorem |5.1| and related results is really necessary, as an example due to Walker 
shows. The example, found in Wal91 , is of a C 1 -diffeomorphism / of the sphere, 
which has an invariant open simply connected set U such that p(f, U) is irra- 
tional, but d U contains a circle of fixed points. Naturally, the example is not 
9-nonwandering; in fact d U is a basin boundary (so there is a closed disk B C U 
such that f(B) C intB and dU = f]U\ f~ n (B). This implies that the backward 
orbit of sufficiently small cross-sections remains arbitrarily close to d U, thus failing 
to be (9-nowandering. 

This example also shows that the theorem fails if one weakens the definition of 
9-nonwandering to require only that for some compact K C U, the set U\K does 
not contain the forward orbit of any wandering cross-section (i.e. one really must 
require that forward and backward orbit of a wandering cross-section leaves U\K). 
Figure fl3] shows what the boundary of U looks like. In this case, U is considered 
to be the unbounded connected component of the complement of the continuum 
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depicted (together with the point at oo). The boundary of U consists of the circle 
C of fixed points (the smaller circle) together with a set of "hairs" accumulating on 
C. The endpoint of each "hair" lies on a larger circle (dotted circle) and forms a 
Cantor set which is invariant by /; the dynamics on that Cantor set corresponds to 
that of a Denjoy example. However any other point outside d U is repelled towards 
oo. 




Figure 13. Walker's example 



Exa mple 9.2 (Non-contractible boundary). Let /' be the map from Example 
9.1 and U' the open set with p(f',U') irrational. Note that B = U is a closed 



topological disk bounded by a circle of fixed points. We may assume (by using 
an appropriate conjugation) that B is the unit square [0, l] 2 on K 2 . Define a 
map /: T 2 — > T 2 by f(z) = 7r(/'(z)), where z is the element of 7r _1 (z) in [0, l) 2 . 
Then / is a homeomorphism of T 2 , with two circles of fixed points, C\ — 7r(M x 
{0}) and C 2 = tt({0} x K). Let U = ir{U'). Then clearly C x U C 2 C dU, and 
p(f, U) is irrational. Thus we have produced an example on T 2 with an invariant 
open topological disk U with irrational prime ends rotation number and a non- 
contractible boundary. 

9.2. On the hypotheses of Theorem |4.2| Here we give two simple examples 



that show that the number N a „ from Theorem 4.2) really depends on both the 



prime ends rotation number a and the genus of the surface S. 

Example 9.3 (dependence on a). Given N € N, N > 1, let / be the rotation by 
a = 2irN, on the sphere C U {oo}, and let U be the unit disk. Clearly, U is /- 
invariant and its prime ends rotation number is equal to 1/iV. Then the boundary 
of U contains an A^-translation arc, so the constant N ai o is necessarily greater than 
N. 

Example 9.4 (dependence on the genus). Let N be a positive integer. We intro- 
duce an area-preserving homeomorphism / on a closed surface S (of genus n = 2N) 
that leaves invariant a simply connected open set U, and admits an TV-translation 
arc outside any given compact set K C U but intersecting both U and its comple- 
ment, while p(f, U) = 1/2. 

Let Si be the sphere M 2 U {oo}, and let R be the rotation of 180 degrees centered 
at the origin o. One can easily define a volume preserving diffcomomorphism g such 
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that g commutes with R, and o is a hyperbolic fixed point with a homoclinic loop. 
We denote by U, V and V — R(V) the connected components of the complement of 
the homoclinic loop. See Figure [l4j We further assume that there is a ^"-invariant 
smooth closed disk D in V, whose orbit is the union n disjoint closed disks. Let 
D' := R(D), where D the interior of D. (One may define g by using the time-one 
map of the Hamiltonian flow associated to a suitable height function.) 




Figure 14. 

Now, let 71 and 72 be two arcs as in Figure [14] We assume that 71 intersects U 
and its extremal points are a G dV and c € 3D, and the extremal points of 72 are 
R(c) and R(g(a)). We also assume that g l {"lj) are pairwise disjoint for j = 1, 2 and 
i = 0, 1, . . . , n — 1. 

We will make a surgery on Si as it follows. Let W be the g-orbit of DUD' which 
is the union of 2n disjoint disks. Let So — S±\ W, Rq := R\s a , 9o '■= g\s an d 
/o := (R o g)\s - Then go commutes with Rq and fo o R = Rq ° fo — go- Observe 
that dSo is invariant for g and for R. By the assumptions, D is diffcomorphic 
to the closed unit disk D C R 2 by a diffeomorphism r such that r(c) = (0,1). 
Let ipi : dD — > dD, = t _1 o L o t, where L(x,y) = (—x,y) on E 2 . Then 
^i(c) = c. Let tp : dSo — > dSo such that ip(p) = g % o ip x o g~~ l (p) if p € g l {dD) and 
V>(p) = Ro g l oip 1 o g~ % o R^ 1 (p) if p g g l {dD'). It easy to see that on 9So one has 
ipoR = Roip,ip 2 = R 2 = (ifjo R) 2 = Id and ip o g = g o ip. 

Now, let be the quotient So/{R ° ip)- Then, it follows that S is an orientable 
surface of genus n and the homeomorphisms Rq and go extend to the quotient 
surface S, and so does fo. We denote by / the homeomorphism on S obtained 
from fo. It then follows that 71 U 72 projects to an arc 7 (note that c and R(c) are 
identified), which is an iV-translation arc for / that intersects the /-invariant disk 
U and its complement. Also one could have chosen 7 lying outside any previously 
given compact set K C U, and it is easy to verify that p(f, U) = 1/2 (modZ). 

9.3. Optimality of Theorem |6.1| and its corollaries. The next examples show 
that the second case in Theorem |6.1| and Corollary |6.4| cannot be excluded, even 
for smooth area preserving maps. 

Example 9.5 (Smooth example with a fixed point). Let us show that, at least 
for some irrational values of a, there are C°° area-preserving examples where the 
second case holds in Theorem 15.11 
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Let / be a transitive area preserving C°° diffcomorphism on the sphere S 2 ob- 
tained by the Anosov-Katok method so that it fixes only two points, say the north 
and the south poles p, q, and moreover almost every point has an irrational rota- 
tion number a for f\&, where A — S \ {p, q}. Such a diffeomorphism exists (it is 
done explicitly in [Cro06j ; we refer the reader to |FK04j for a general survey on the 
Anosov-Katok method). 

Let D be an open disk containing p and disjoint from a neighborhood V of q. Let 
K be the connected component of [\ n pvf n {D) containing p. Then K is compact, 
connected and invariant. Moreover, K is non-separating, because the complement 
of K contains the orbit of V, which is dense and connected. Thus U = § 2 \ K is 
an open invariant topological disk. 

It is possible to show that K intersects dD. This follows from the results of 
[LCY97) , which imply that if K does not intersect d D then either p is "dissipative" 
(i.e. every neighborhood of contains a loop around that is disjoint from its image) 
or p "saddle-point" (there exists an integer m such that the Lefschetz index i(f m ,p) 
is non-positive). Indeed, p cannot be dissipative because / preserves area, and it 
cannot be a saddle-point because i(f m ,p) = 1 = i(f m ,q). The latter follows from 
the fact that the two indices must add up to 2 by Lefschetz' formula, and none of 
the two indices is greater than 1 by a theorem of Pelikan and Slaminka PS87J (see 
also [LC03] ). 

Finally the fact that p(f,U) = a follows from |LC031 §6], so the second case 
from Theorem |6.1| and Corollary |6.4| holds. 



Example 9.6 (Example with a fixed point and any irrational a). The previous 
example shows that case (ii) in Theorem 5.1 can hold for smooth maps, but it 
due to the technique used in the construction of the example, the number a will 
always be a Liouville number. It is possible that no such example exists if a is 
Diophantine and / is smooth enough (due to a KAM-type phenomenon). However 
if one drops the smoothness condition, one may produce homeomorphisms with the 
same properties as the previous example, for any given irrational a G R/Z. 

We sketch a way to obtain such example using the Denjoy-Rees technique from 
BCLROJj- As mentioned in section C2 of [BCLR07 , the techniques of said article 
allow to obtain for any a € K \ Q a homeomorphism / : § 2 — > § 2 which is semi- 
conjugate to a rigid rotation by angle a of the sphere (fixing the north and south 
pole) via a surjection h: § 2 — > § 2 which leaves the equator C invariant, such that 
there is a vertical segment I that touches the equator at its base and is mapped by 
h to a single point on the equator, diam(/ (£)) — > as k — > ±oo and h is injective 
outside the orbit of I (see Figure 15 ), so that A = C U Ufcez f k (^) * s an /-invariant 
connnected set. Let U be the upper connected component of § 2 \ A. Then f\u 
preserves a non-atomic Borel probability measure of full support (obtained by 
pull-back of the normalized Lebesgue measure on the upper half-sphere via h), and 
using the monotone semi-conjugation h it is easy to verify that p( f, U) = a. 

If we collapse the circle C to a single point p , we obtain a surface S' ~ S 2 , a 
homeomorphism 5' — > S' for which p is a fixed point, and an /'-invariant set 
A' which contains pa and is the boudnary of U in S'. Furthermore, by Corollary 



3.9 we have that 



p(f, U, S) = P (f, U, S \ {p }) = p(f, U, § 2 \ C) = p(f, U, S 2 ) = a 
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Figure 15. A before and after collapsing C 

The measure // induced by \x in S is a non-atomic Borel probability measure of full 
support, and by the Oxtoby-Ulam theorem [OU41j we may choose a homeomor- 
phism cf>: S — > S such that /" = (frfcf)' 1 preserves Lebesgue measure, so /" has the 
required properties. 

10. Appendix: Proof of Theorem 13.31 

10.1. Proof of parts (1) and (3). We begin by proving (3). First observe that 
any chain C of U as a subset of S is equivalent to some chain C = (Z)-)i 6 N such that 
djj D[ is a cross-cut of U with endpoints in S : this is easily done by perturbing 
the endpoints of the cross-cuts defining the given chain, using the fact that S \ So 
is totally disconnected (we leave the details for the reader) . Thus a chain of U as a 
subset of S is always equivalent to some chain of U as a subset of So- If, in addition, 
C is a prime chain of U as a subset of S, then it follows from the definition that C 
is a prime chain of U as a subset of S . 

There is a natural way to define an extension : Cf(C/, S) — > C£(U, So) of the 
inclusion i: U — > C£([/, So), as follows: Let p g bf(J7, S) be a prime end, and let C 
be a representative prime chain. By the previous observation we may choose C so 
that it is also a prime chain of U as a subset of So- If we let i*(p) be the prime end 
in be(U, S ) represented by C, then i*(p) does not depend on the choice of C, but 
only on the choice of p, so it is well defined. Thus, letting i*(z) = z for z G U, we 
have defined a map i* : C£(U, S) —> C£(U, So)- 

To show that i* is surjective, one needs to verify that if C is a prime chain of U 
as a subset of So, then it is also a prime chain as a subset of S (for that would imply 
that the prime end corresponding to C in b£(U,S) is mapped by z* to the prime 
end corresponding to C in b£(U,S ))- But this is true, because if C is divided by 
some chain C of U as a subset of S, then from the remark at the beginning of the 
proof, C is equivalent to some chain C" of U as a subset of So, and so C" divides 
C. But the latter, together with the fact that C is a prime chain of U as a subset 
of So, implies that C divides C" . Since C" is equivalent to C, we conclude that C 
divides C , showing that C is indeed a prime chain of U as a subset of S. Therefore 
i* is surjective. 

To see that i* is injective, note that if p and p' are elements of bj (U, S) such that 
i*{p) = i*(p'), the definition of i* implies that p and p' are represented in b£(Z7, S) 
by chains C and C (respectively) which are also prime chains of U as a subset of 
So representing the same prime end = i*(p') in bf (U, So). This means that C 
and C are equivalent, and so p = p'. 




46 



ANDRES KOROPECKI, PATRICE LE CALVEZ, AND MEYSAM NASSIRI 



For the continuity of i*, it suffices to show that if D is a cross-section of U as a 
subset of So then i~ 1 (DU£ u,s (D)) = D\J£jj i s{D) (recalling that £ u ^s(D) denotes 
the prime ends in bg (U, S) which are represented by some chain that divides D). 
But this is clearly true, since as we mentioned at the beginning of the proof, any 
prime chain C of U in S is equivalent to some prime chain C of U in So, which can 
be found by perturbing the cross-cuts defining the elements of C, and if C divides 
D then C divides D as well. 

To conclude that i* is a homeomorphism, let us show that i~ l is continuous. The 
fact that every element of bs(U,S) is represented by some prime chain of U as a 
subset of So implies that if D is a cross-section of U as a subset of S then £jj,s(D) 
is the union of all sets of the form £jj,s(D') where D' C D is a cross-section of 
U as a subset of So- Since we already showed that i*(£u,s(D')) = £u,s {-D') for 
such cross-sections, we have that i*(D U £(D,S)) is a union of sets of the form 
D' U £u.S (D') where D' C D is a cross-section of U in So- Thus i* maps open sets 
to open sets, and we conclude that i* is a homeomorphism, completing the proof 
of (3). 

Finally, we observe that for the case that U is relatively compact in S, part 
(1) is already known; see for instance |Mat82] or |Mat81j . In particular, since 
Ci S = SUbi S is compact and bi S is compact and totally disconnected, applying (3) 
to the closed surface Ci S in place of S we conclude that cs(U, S) is homeomorphic 
to cs(U, Ci S), from which (1) follows. □ 

10.2. Proof of part (2). Note that, since parts (1) and (3) are already proved, 
we may use Proposition |3.4[ which only depends on those items. We begin by 
observing that since U C So, the boundary of U in So has more than one point 
(the only case where ds U can be a unique point is when So is a sphere, which is 
impossible in our case). Thus cg(U, So) is well defined. 

Claim. Any prime chain ofU as a subset of S divides some chain which is a prime 
chain of U as a subset of So- 



Proof. Observe that in view of parts (1) and (3) of Theorem 3.3 we may assume that 
S is a closed surface, replacing S by Cj S and S by S \ b t S (the hypotheses still 
hold because bi S is closed and totally disconnected in Ci S). Thus, by Proposition 



3.4 any prime end in bs{U, S) is represented by a prime chain C = (Di)^ such 
that djj Di — > x in S for some x g d U . Hence, to prove our claim it suffices to 
show that C divides some prime chain C of U as a subset of So- 

Suppose first that x G So- Then, since du Di — >• x, there is io € N such that 
9[/ Di C S if i > io, and so C — (Z)j)j>j is a chain for U as a subset of S as well. 
The fact that C is a prime chain of U as a subset of S also implies that it is a prime 
chain of U as a subset of So, and so our claim holds with C =C. 

Now suppose that x £ S \ So, and write S' — Ci(So) and — du Di. Since 
ji — » x in S as i — > oo, it follows that for any neighborhood W of bi(So) there 
is io such that 7, C W if i > io- By compactness of S', if we replace (Di) ieN by 
a subsequence (which is a chain equivalent to C), we may assume that (cls> Ji)iefi 
converges to some subset of S' in the Hausdorff topology, which must be compact 
and connected, because so is c\s> 74 for each i e N. Moreover, by our previous 
observations, the Hausdorff limit of such a sequence must be a subset of bi(So), 
which is a totally disconnected set. Thus the limit is a single point; i.e. there is 
p G bi(So) such that 7, — > p in S' as i — > 00. 
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Let (i?i)i e pj be a sequence of closed topological disks in S' such that -Bj+i C int Bi 
and f\ i<£N Bi = {p}. Since bi(So) i s totally disconnected, one may choose each Bi 
such that ds' Bi n bi(So) = 0, so d Bi = ds 1 Bi C S . Since ds' U is connected 
(because S' is compact), it follows that d Bi<lds> U ^ for each i (if we assume that 
U is not entirely contained in B± , which we can) . Choose a connected component "f[ 
of d B\ H [/, and note that 7j is a cross-section of U in Sq. Since C is a prime chain 
of U in S, we can find ii € N such that 7^ ^ -D^. Moreover, if i\ is large enough 
then d D i± — 7^ C B\, and so it is disjoint from 7^, implying that 7J C 17 \ 7?^. 
Thus the connected component D[ of U \j[ containing 7^ also contains D^. 

Choose 12 > %\ large enough so that 7i 2 C Bi- Since d B2 D U separates ji 2 
from 7j in U, it follows easily that there is a connected component 72 of 9 £?2 n U 
that separates 7, 2 from 7^ in {7. Since Di 2 C D[, one deduces that the connected 
component D' 2 of U \ j' 2 containing 7 i2 satisfies D i2 C D' 2 C D[. Repeating this 
process recursively, we obtain a chain C = (D0jg^ of U in So and an increasing 
sequence (ik)k&i such that djj D' k C d Bk and /J ifc C D' k for each fe G N. This 
means that C regarded as a chain of U in S is divided by C. Moreover, since 



djj D' k C Bk, we have that 9(j D' k — ¥ p in S' = ci(Sq) as fe — > 00, so Proposition 3.5 



implies that C is a prime chain of {7 in So, completing the proof of the claim. □ 

Given p £ b£(U,S), let p' be a prime end in bf(i7, So) such that some prime 
chain C = (Dj)j e [sj of t/ in S* representing p divides a prime chain C — (D'^i^ of 
U in So representing p' . The previous claim guarantees that such p' exists. We will 
show that it is uniquely determined by p. Suppose on the contrary that there is 
p" € bf(i7, So) represented by a prime chain C" = (D")^, such that C divides C", 
and p" ^ p' . Observe that cl C£ ms ) D[ is eventually contained in any neighborhood 
of p' in cs(U,So) ~ D, and the analogous fact for p" . This implies that D\ and 
D" are disjoint if i,j are chosen large enough. But this is a contradiction, because 
both C and C" are divided by C, so for any i,j£N the set n D" contains some 
element of C. This contradiction shows that p' is uniquely determined by p, and so 
letting i*(p) — p' we have a well-defined map i* : hgQj, S) — > hs(U, So). 

Moreover, if p' e bf([/, So) is represented by a prime chain C = (7?9«eN, then 
i* 1 {p') consists of all prime ends represented by a chain of U in S that divides C, 
which is precisely 

i- 1 (p') = [}£u,s{D' i ). 

This is a nested intersection of open intervals, and since the endpoints of du D[ in 
S are disjoint from the corresponding endpoints of djj D' i+1 (from the definition of 
chain), it follows (by the remarks after Proposition |3.5[ ) that the endpoints of djj 
in ce(U,S) are disjoint from the corresponding endpoints of djj D i+ i in cs(U,S). 
This means that the closure of £u t s(D' i+1 ) in b £ (U,S) is contained in £ u ^(D' i ) 
for each j £ N. Therefore the intersection definining i^ 1 (p / ) is a nonempty closed 
interval, showing that i* is surjective and monotone. 

Define i x (x) — x for x e U, so that i* : cs(U, S) —> cs(U, So) is now a monotone 
surjection extending the inclusion. To see that i* is continuous, it suffices to show 
that C 1 (7? U £jj,s {D)) = D U £u t s{D) for each cross-section D of U in So- Since 
i~ x (D) = D, it suffices to show that i~ 1 (£u,s {D)) = £u,s{D). Moreover, it is clear 
that i~ x {£ u,S (D)) C £u,s(D), so we need to prove that i*(£u,s(D)) C £u,s„(D). 
Let p € £u t s(D), so that p is represented by some prime chain C = (Di) ieN of U 
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in S that divides D. Let C' = (Dj)jgN be a prime chain of U in So that represents 
i*(p) (so that it is divided by C). We will show that C also divides D, which is 
enough to complete the proof since it implies that i*(p) € £u,s {D). 

By Proposition 3.5 we may assume that there is x € ci{Sq) such that djj D' t — > x 
in Ci(5'o) as i — > oo. If x £ So, then the same proposition implies that C is a prime 
chain of U clS clS £1 subset of S, and so it is equivalent to C, proving that it divides D, 
which is what we wanted. Now suppose that x G bi(S'o). Since els djj D c So, the 
fact that djj D[ — > x as i — > oo implies that djj D[ is disjoint from djj D if i is large 
enough. On the other hand, since C is divided by C which in turn divides D, it 
follows that D\ n D ^ for all i. From these two facts, we have that either D\ C D 
for some i £ N, or D C for all z 6 N. But in the latter case, the endpoints of 
djj D in cs(U, So) are accessible prime ends that divide D[ for each igN (and so 
they divide C), contradicting the fact that C is a prime chain of U in So- Thus 
D[ C D for some i £ N, showing that C divides D. This completes the proof of 
Theorem [3~3| (2). □ 
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